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EXECUTIVE SUMMARY 


Department of Defense (DoD) use of Earned Value Management (EVM) program 
control tool has significantly increased in the last ten years. DoD acquisition policy and 
training promotes EVM as a cost and schedule management tool, tracking the earned 
value of the work completed per the baseline plan. Acquisition Category ID programs 
like the US Air Force F-22 fighter program use EVM to manage their software 
development efforts, but has the program’s implementation of EVM followed the 
industry-recognized 32 criteria outlined in ANSI/EIA-748-A-1998 Earned Value 
Management System Standards and necessary to successfully implement EVM. 

Using these 32 criteria, an evaluation was performed aimed at assessing the 
implementation of EVM in the F-22 program. The goal: to academically appraise the 
program’s use of EVM in managing Spiral 2, an F-22 avionics software modernization 
effort. To accomplish this goal a detailed evaluation of how the program meets the 32 
criteria was conducted along with analysis of program data, interviews of subject matter 
experts and a statistical questionnaire conducted with F-22 personnel. Results indicated 
areas of possible improvement in the use of EVM and potential changes to the F-22 
development environment to improve planning, scheduling and budgeting of an EVM 
baseline. A concise description of the results for each of these assessments follows 
below.! 

The subjective evaluation of the 32 criteria exhibited several areas of interest. 
First, 22 of the 32 data points (68.75 percent) either met or exceeded the intent of their 
respective ANSI criterion by earning an excellent or satisfactory rating. Of the remaining 
data points, six (18.75 percent) exhibited either a marginal or insufficient rating, while 
four (12.50 percent) exhibited an inconclusive rating (an inconclusive rating resulted from 
insufficient supporting data). Major factors contributing to less-than-satisfactory ratings 
included usage of undefinitized contracts and an observed lag between cost data 


collection and reporting. 


1 Please refer to Chapters III and IV for a more detailed discussion of these findings. 
1 


The questionnaire, which reached 100 percent of government and contractor 
personnel dedicated to Spiral 2 (the first F-22 avionics software upgrade since Initial 
Operational Capability (IOC)), exhibited the following: 


e Personnel had a slight majority opinion EVM has some value 
° Personnel had a slight majority opinion EVM has some usefulness 


e Most personnel are not aware of the ANSI/EIA-748-A-1998 EVMS 32 


Criteria 


Additional statistical analysis of the questionnaire concluded that: 


° Higher EVM value resulted in more EVM usage 
e Higher EVM usefulness resulted in more usage of EVM in managing 
e Higher EVM knowledge did not result in higher EVM value 


Finally, the interviews independently verified the separate assessment of the 32 
criteria and served to explain the results of the questionnaire. 

The summation of the 32 criteria evaluation, questionnaire analysis, and 
interviews led to the conclusion that the F-22 Program Team, with respect to its software 
development efforts, did meet the intent behind the industry standard ANSI/EIA-748-A- 
1998 with respect to most of the criteria. There were, however, several criteria that were 
found to be marginal or insufficient. Research identified several areas of incompatibility 
with regard to EVM and software development (or any other similarly dynamic 
environment) that may preclude functional managerial controls. Specifically, as software 
development efforts progress beyond early stages, less-defined tasks become more 


difficult to manage via EVM. 


I. INTRODUCTION 


A. BACKGROUND 


1. Earned Value Management 


DoD Program Managers’ use of EVM as a measure of program cost and schedule 
performance has significantly increased in the last ten years. DoD acquisition policy and 
training promotes EVM as a tool for measuring program health by tracking the “earned 


value” of the work completed per the baseline plan. 


Facilitating an intelligent discussion concerning EVM first requires a rudimentary 
knowledge of the EVMS as it exists today. Consequently, this paper offers the following 
primer for those either new to or requiring a refresher in the basics of EVM. Proposed 
methods for evaluating an EVMS and a discussion of the ANSI/EIA-748-A-1998 EVMS 


criteria follow the primer. 


a. An EVM Primer 


‘ 


Earned Value Management: “...a tool for effectively integrating cost, 
schedule and technical performance management” (DAU, 2005). To integrate said cost, 
schedule, and performance involves making those measurements visible. The core of that 
visibility revolves around three measurements (DAU, 2005): 

e Budgeted Cost of Work Scheduled (BCWS) — This measurement sums 
the budgets for all work scheduled for accomplishment—including in- 
process work—plus the amount of apportioned effort scheduled for 
accomplishment at a specified point in time. The BCWS value at project 
conclusion equates to Budget at Completion (BAC). Alternate 


terminology includes “planned value” and the Performance Measurement 


Baseline (PMB). 


e Budgeted Cost of Work Performed (BCWP) — This measurement 


provides the value of work actually performed and uses budgeted costs to 


calculate the cost at a specific point in time. Also known as Earned Value 


(EV). 


e Actual Cost of Work Performed (ACWP) — This measurement provides 
the costs actually incurred and recorded (as opposed to budgeted costs) in 
accomplishing the work performed at a specific point in time; normally 
the contracted organization provides ACWP data directly. Often it is 
simply called Actual Cost (AC). 


Program Managers use comparisons of these three measurements to gauge 
a contractor’s progress against an initially agreed-upon baseline (as a rule, the PMB). 
For instance, a quick comparison of BCWS and BCWP provides a useful measurement of 
Schedule Variance (SV).° An intuitive analysis reveals that when work performed 
exceeds work scheduled for a given program, the program in question is ahead of 


schedule. 


Likewise, a quick comparison of BCWP and ACWP provides an equally 
useful measurement known as Cost Variance (CV).° At an elementary level, when actual 
costs exceed budgeted costs within a scrutinized program, that program warrants a “cost 
overrun” classification. Armed with CV and SV, a program manager and his or her 
Integrated Product Team (IPT) can now calculate a rudimentary Estimate at Completion 


(EAC). 


For those not familiar with the term, EAC answers “What do we now 
expect the total job to cost?” (Haupt, 2002). Take an original EAC of 20 and a one time 
CV of -4. Subtract this 4 point overrun from the original EAC of 20 (1.e., 20 — (-4)), and 
the new EAC reflects a projected cost of 24. The basics of EAC now explained, the next 


level of analysis involves creating indices for both cost and performance measurements. 


Within the EVM body of knowledge, these indices are known as Cost and 
Schedule Performance Index (CPI and SPI).4 The CPI indicates cost performance 


2 SV =BCWP - BCWS 
3. CV=BCWP - ACWP 
4 CPI = BCWP / ACWP 


efficiency related to work the contractor has actually accomplished at a specific point in 
time. In other words, it provides a measurement of the value of work the program 
receives from every dollar given towards the effort. For example, assume BCWP = 8 and 
ACWP = 10. The resulting CPI (.8) reports that every dollar invested into the project 
results in .80 cents of effort. CPI’s ultimate use (assuming a constant CPI) stems from 
enabling government IPTs to project the final cost of a contract and even determine the 


likelihood that the contractor can recover (Heise, 1991, p. 95). 


Along those same lines, SPI indicates schedule efficiency at a specified 
point in time. For example, if BCWP = BCWS, then SPI = 1. An index of 1 indicates 
that the supplier is performing on schedule whereas an index of 1.1 indicates an ahead-of- 
schedule condition (Smith, 1977). Like CPI, SPI’s ultimate utility stems from bestowing 
the ability to project the final completion date of a contract alongside the probability that 
the contractor can meet or beat the original project completion date, given a reliable 


budget. 


This segment represented a very basic working knowledge of EVM; the 


next topic concerns evaluating the effectiveness of EVM within a given program. 


b. Evaluating the Effectiveness of an EVMS 


When, in 1995, the National Defense Industrial Association (NDIA) 
decided it was too unwieldy and expensive to abide by DoD’s 35 Cost/Schedule Control 
Systems Criteria (C/SCSC), industry leaders took initiative and developed EVMS, which 
remains more or less in its same form today.’ Little more than twenty months later, the 
32 guidelines from the new industry standard, American National Standards Institute / 
Electronic Industries Association (ANSI/EIA) 748, became the DoD baseline for EVMS, 
as well. Essentially, ANSI/EIA-748-A-1998 states how to effectively apply earned value 
concepts that will aid in successful program management (Fleming & Koppelman, 2000). 


Even knowing that, one might ask, “So what?” 





SPI = BCWP / BCWS 


To begin with, history has shown that no single EVMS can hope to meet 
every need (management, reporting, etc.) with respect to performance measurements. 
Differences in programs, as varied and unrelated as organizations, weapon system 
architecture—even how well government and contractor teams interact—make it 
unrealistic to prescribe a one-size-fits-all approach to cost and schedule controls 


(Johnson, 2006). 


Instead, the EVMS Guidelines provide the basis for determining whether 
contractors’ EVM systems meet standards. These broad guidelines serve a two-fold 
purpose (Scott, 2005). First, they allow for common sense applications (read: flexibility) 
on both sides of the table—government and contractor. Second, their comprehensive 
nature reassures the government that with each report it receives reasonably reliable 


performance data. 


That first purpose leads to the conclusion that common sense should rule 
the design, employment, and subsequent iterations of a program’s EVMS. Unfortunately, 
many times government standard operating procedures and common sense have not 
positively correlated. More often than not, standard operating procedures meet the letter 
of the guidelines, but not their intent. | Lacking support for intent, the resultant 
incongruence almost always eventually fails to support management's needs, and the 


EVMS inevitably fails as a management control system (Scott, 2005). 


The second purpose depends heavily on one assumption in particular: the 
contractor possesses effective internal controls (Scott, 2005). The presence of these 
controls makes reliable reporting possible. Without that foundation, even the most 
stringent process attempting to follow ANSI/EIA-748-A-1998 fails proper 
implementation due to uncertainty surrounding the cost and schedule figures produced by 


the contractor. 


EVM systems that comply with the intent and nature of the guidelines 


facilitate project work scoped in its entirety, to include detailed planning. Properly 


5 Fleming and Koppelman assert that industry leaders perceived the previous DoD- 
driven standards as non-user-friendly and incompatible with the needs of private industry. 
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implemented EVM systems also facilitate full integration of cost, schedule, and project 
performance objectives into a performance measurement baseline against which actuals 
(work, cost) can be measured. An effective EVM system tailors itself to a given program 


based on a foundational baseline that fosters full and / or appropriate control. 


From a reporting standpoint, an effective EVMS uses and provides 
information that utilizes the defense industry standard Work Breakdown Structure 
(WBS), which delineates product work packages as well as organizational responsibility. 
Within each WBS, quantifiable measurements of metrics—to include SV, CV, SPI, CPI, 
etc.—should generate at the lowest appropriate organizational levels where the actual 
work occurs. From those fundamental organizational levels up through the highest levels 
of management, the EVMS should reflect strong discipline in reporting. Otherwise, 
management “dashboards”, or high-level reporting to upper management used for 


decision-making, would prove ineffective. 


From a management utilization standpoint, an effective EVMS provides a 
virtually on-demand analysis of significant variances (e.g., SV, CV) along with narrations 
of forecasted impacts. An effective EVMS becomes a key enabler of management 
actions that may mitigate risk, manage cost, and manage schedule. For example, the 
development of iterative estimates of final contract costs, beginning with the initial BAC 
and ending with the last Latest Revised Estimate (LRE), rates as both management 
control and risk moderator. Effective EVM systems impart upon a program at least a 
modicum of visibility into subcontractor performance, performance that directly affects 


the prime contractor (Scott, 2005). 


Having seen the benefits of proper guideline adherence, and how these 
yet-to-be-defined guidelines evaluate an EVMS, the next section contains an overview of 


the ANSI EVMS guidelines. 


Cc. The ANSVEIA-748-A-1998 EVMS Criteria 


Although ANSI’s EVMS Criteria may have changed in name and wording 


over recent years, their intent has remained largely unchanged since their inception. 
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“Each criterion addresses a major principle necessary for effective management of large, 
flexibly priced defense projects...criteria are often described as common-sense 
management practices that any well-managed defense contractor would use” 


(Christensen, 1998). Table 1 provides a concise list of the 32 criteria. 


Criteria 


Group 1: Organization 
Define authorized work (WBS elements) 
Identify organizational responsibilities 
Integrate the system 
Identify overhead management 
Provide for performance measurement 


Schedule the work 
Identify products, milestones and indicators 
Plan the Performance Measurement Baseline (PMB) 
Establish budgets for work 
Identify work packages 
Summarize work package budgets to control accounts 
Identify and control level of effort 
Establish overhead budgets 
Identify management reserves and undistributed budget. 
Summarize budgets to target cost 
Group 3: Accounting 
16 Record direct costs 
17. Summarize direct cost to the WBS 
18 Summarize direct cost to the organization 
19 Record indirect costs 
20 Identify unit/lot costs 
21 Record material costs 
Group 4: Analysis 
22 Identify schedule and cost variances 
23 Analyze schedule and cost variances 
24 = Analyze indirect costs 
25 Summarize data elements and variances for reporting 
26 Implement managerial actions 
27 Develop revised estimates of cost at completion 
Group 5: Revisions 
28 Incorporate changes into plans, budgets and schedules 
29 Reconcile budgets changes 
30. ~=Control retroactive changes 
31 Control revisions to the program budget 
32 Document changes to the PMB 
Table 1. | ANSI/EIA-748-A-1998 EVMS Criteria 





As presented within Table 1, ANSI/EIA-748 organizes the criteria into the 
following five areas based on major project management activities: 


° Organization 


e Planning, Scheduling and Budgeting 


e Accounting 
e Analysis 
e Revisions 


The Organization area (five criteria total) covers the definition of 
authorized work within a program. It also tasks program planners to ensure some 
effective delineation of organizational structure and their respective responsibilities. 
Finally, this area stipulates some integration of the program’s work with the 
organizational structure that enables effective and meaningful measurements of cost and 


schedule performance. 


The second area, Planning, Scheduling and Budgeting (ten criteria total), 
contains information regarding proper planning, scheduling and budgeting of authorized 
work so information gleaned from the system remains meaningful. Specifically, this area 
supports and explains the ideas of task interdependency awareness; milestones, delivery 
criteria, and other measures of progress; and benefits of stable and measurable units. It 


also references some areas of customer interaction. 


The Accounting area (six criteria total) includes a discussion of 
maintaining accounting discipline so information remains comparable from reporting 
period to reporting period. Not surprisingly, this area also discusses direct costs, indirect 
costs, and unit costs as they pertain to a formal EVMS. Additionally, this area discusses 


the integration of a material accounting system with the planned EVMS. 


The next area, Analysis (six criteria total), suggests the frequency of 
submitting EVMS reports (at least monthly) and what basic data to include (ACWP, 
BCWS, BCWP, SV, CV). It also presents customer reporting requirements and pre- 
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requisites for delivery of a meaningful management control product. This area also 


examines implementing changes based on important identified variances. 


Lastly, Revisions (five criteria total) discusses how EVM practitioners 
should incorporate changes to reports, thereby enabling timely and effective changes to 
an affected program. This area also explains the difference between appropriate changes 
(e.g., correction of errors) and inappropriate changes (e.g., hiding flawed information). 


Revisions sets forth that practitioners should always document changes. 


With the concept of EVM explained, the next section contains an introduction to 


the F-22 program. 


2. The F-22 Program 


In the summer of 2002, F-22 System Program Director (SPD) Brigadier General 
William J. Jabour confirmed what other program officials had cautiously hinted at for 
several months: the F-22 would miss its scheduled start date for the program's Dedicated 
Initial Operational Test and Evaluation (DIOT&E) (Chapman, 2002). With internal 
pressure from Air Combat Command already mounting and external pressure in the form 
of congressional involvement imminent, the timing could not have been worse for the 
forecasted six month schedule slip. Senior DoD leaders demanded to know how such a 
monumental program failure could occur without warning and talked of ominous 


consequences should another slip transpire. 


Regardless of these pressures to maintain schedule, a fact-finding group known as 
the Red Team (assembled by concerned program proponents) arrived at a threatening 
conclusion: the program would slip again. The situation reached critical mass during the 
close of 2002, when Secretary of the Air Force, Dr. James G. Roche, reassigned Generals 
Jabour and Shackelford in favor of “new leadership...necessary to achieve the Air 
Force’s objectives” (Air Force Print News, 2002). Internally, Air Force leadership 
wondered how this could happen, since the program performance measurement practiced 
by the F-22 program (i.e., Earned Value Management) should catch impending cost and 


schedule problems early in the process. 
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In fact, three years earlier the Government Accountability Office (GAO) found 
that the F-22’s prime contractor, Lockheed Martin, retained reports exhibiting a 
downward trend with respect to its accomplishment of planned work.6 Specifically, 
software development (i.e., avionics’ Operational Flight Program (OFP)) for the F-22 fell 
behind to such an extent that a rebaselining of schedule occurred on August, 1998. 
According to the GAO (1999), the causes included: avionics development falling behind 
schedule, unrealistic avionics schedule goals, and the critical nature of avionics with 


respect to the weapon system. 


This schedule slip for one of DoD’s largest weapons system acquisitions joined a 
long list of timeline adjustments to the right. Numerous program slips occurred over the 
F-22’s twenty-plus years of development. Figure 1 shows F-22 program milestones 
compared to other legacy aircraft development efforts. The cumulative effect of all F-22 
schedules slips resulted in the program taking “76 percent longer than estimated to 
achieve first flight and 57 percent longer to reach first production [and] 19 percent longer 
to reach Initial Operational Capability IOC)” (Younossi, Stem, Lorell, & Lussier, 2005). 
One driving factor is persistently manifested behind each and every scheduling failure: 


avionics development. 


6 As of June 1998, LMT estimated planned work not completed at $115M. 
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Figure 1. | F/A-22 Schedule Slippage Is Higher Than the Historical Average 


(From: Younossi et al., 2005) 





Officially designated today as the F-22A, the program (formerly known as the 
Advanced Tactical Fighter, F-22, and F/A-22) has undergone dramatic mission 
requirement changes since its inception in the late 1980’s. Originally, US leadership 
envisioned the F-22 as an answer to the Soviets’ Su-27 and MiG-29 aircraft that 
threatened to technologically usurp the global dominance of the Boeing F-15 fleet. 
However, in the eyes of budget hawks the fall of the Soviet Union (and subsequent 
termination of the Cold War) changed the requirement for a next generation air-to-air 
platform. Many questioned the need to move away from the historical dominance of the 
current F-15 fleet in a world lacking adversarial nation-states with upgraded, current 


generation fighter aircraft. 


This new global reality forced the Air Force to move away from a strict air-to-air 
role and instead introduce air-to-ground requirements to the F-22 program. While the 
advanced air-to-air capabilities of stealth, supercruise, and integrated avionics remained 
the foundation of the revolutionary fighter, the program solidified plans that would 
incrementally add additional capabilities such as delivery of Joint Direct Attack 
Munitions and enhanced air-to-ground radar to the jet. As a result, the Air Force directed 


the F-22 System Program Office (SPO) to initiate a new Modernization program, with its 
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main objective being the development of in-line and post-production upgrades to the 
fighter. Though joint SPO and Contractor Team plans to integrate a majority of these 
new requirements looked to post-EMD, several key requirement changes required 
introduction and implementation prior to completion of both EMD and approval to 


proceed with Full Rate Production (Younossi et al., 2005). 


While many defense experts typically point to the introduction of these new 
requirements as a key driver behind the schedule slips and accompanying cost over-runs, 
others note that the F-22 experienced significant cost and schedule variances prior to 
introduction of these additional requirements. For example, according to a 2006 study 
by Younossi et al., “[F-22] cost growth was mainly the result of design challenges in the 
airframe (arising from stealth requirements), the integrated avionics suite, and the new 


” 


propulsion system.” Figure 2 highlights this statement by presenting the cost growth of 


the F-22 by major system from 1995 through 2002. 
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Figure 2. F-22 Cost Growth Trends for Major Systems 
(From: Younossi et al., 2005) 


Looking at percentage of cost growth alone rates the F-22’s integrated avionics 


suite as one of the areas of greatest concern. The highly complex avionics subsystem, 
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touted as one of the three key technological advancements contributing to the dominating 
existence of the F-22, certainly explains that cost growth.7 After all, advancements such 
as those fielded in the avionics suite do not come without an appropriate price tag. In 
fact, government experts calculated this subsystem consumed one-third of the F-22 
program budget—“more than any other subsystem, including the airframe” (Younossi et 
al., 2005). This still leaves the question of “What happened that caused such price 


growth?” 


One answer may lie in a comparison of the historical development of avionics 
systems and the F-22’s systems. While legacy aircraft avionics followed a federated 
construct where each avionics subsystem (e.g., Communications / Navigation / 
Identification, Electronic Warfare, Radar, etc.) provided information to the pilot 
independently from other subsystems, the F-22 uses a central core processor to fuse this 
information from the various sensors and other components to present an integrated 
picture to the pilot. This requires extremely large numbers of instructions per second— 
millions for data processing and billions for signal processing—creating extensive 
demands on aircraft computing systems that resulted in significant system lock-ups 


during developmental testing (GAO, 2004). 


In addition to the extreme demands on avionics hardware, the F-22 requires 
software complexity at an unprecedented level to manage the data flowing through said 
hardware. Software designers answered that need with the F-22 Operational Flight 
Program. But once again, increasing complexity and requisite robustness became another 
driver that increased avionics cost. Between October 1993 and April 2000 the F-22 
Software Lines of Code (SLOC) grew approximately 34 percent (Younossi et al., 2005). 
According to the GAO, this SLOC growth, largely driven by requirements and design 


changes, resulted in delayed software deliveries, impacting program cost and schedule, 


7 The other two F-22 “first look, first shot, first kill” capabilities identified as “firsts” 
in US military aircraft are supercruise and stealth. The powerful F-119 engines and 
airframe design provide supercruise capability, enabling the F-22 to cruise at supersonic 
speeds without the use of afterburners. ‘Althoush other aircraft have fielded stealth 
emoloey; the F-22 is noted as being the US first all-weather, “24-7-365” stealth tactical 

ighter. 
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and “accounted for 37 percent of the critical problems reports leading to avionics 


shutdowns in the [F-22].” (2004) 


Conventional wisdom supports the idea that the Air Force—not to mention other 
services—would like to avoid similar situations in the future. Aiding that desire became 
the ultimate driver behind this study. The next section, Purpose and Significance of 


Study, further explores the motivating forces behind this research. 


B. PURPOSE AND SIGNIFICANCE OF STUDY 


Within the context of this study, two goals remained paramount and thus defined 
the project’s purpose. The first goal: determine if current EVM implementation within 
the F-22’s Modernization program will likely prevent an abrupt schedule slip and certain- 
to-follow cost overrun—similar to what occurred during EMD. The second goal: make 
meaningful recommendations, where appropriate, with the objective of strengthening the 
current EVM system implemented to measure the F-22 program’s software development 


performance. Again, these goals delineate the study’s purpose. 


The significance of the study relied upon the expectation of ever-increasing 
weapon system complexity, especially within the realm of software integration. For 
example, upon completion of a Program Office Estimate (POE) on March of 2003, the 
government predicted F-22 EMD costs to exceed Milestone II estimates by 33 percent. 
Granted, a program spanning an inordinately large number of years should expect to see 
mission requirement and technology changes that drive schedule and cost impacts, 
however, the F-22 actually fared worse than the average of other similar development 


efforts, with a schedule growth factor of 1.33 versus a 1.22 average (see Figure 3 below). 
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(From: Younossi et al., 2005, p. 10) 


As evidenced within the first section, Background, avionics development rated as 
a significant cost driver. When considered concurrently with the schedule trials 
experienced during EMD, these facts escalated the importance of discovering whether or 
not the EVMS implemented-of-late worked to minimize the schedule and cost risks of an 
unprecedented software effort. More importantly, given the increasing cost of warfare- 
dominating technology, DoD should address this prior to reaching comparable phases of 
development within upcoming flagship acquisitions: the Joint Strike Fighter and the 
Future Combat System. According to the GAO, these two programs are projected to cost 
DoD over $330B — more than five times the total cost of the F-22 program and roughly 
80 percent of DoD’s entire FY07 Appropriation Bill (2006). 


Thus, learning from mistakes of the past becomes vital within DoD’s resource- 
constrained environment. Seeking significant answers lies in asking pointed questions. 
First, if the F-22 program office practiced and used EVM, how did these problems appear 
to catch everyone (but program detractors, of course) by surprise? Lastly, why weren’t 
the abrupt cost overruns and schedule failures, that cost an SPD his job, caught earlier? 
The next section, Research Questions, further explores and refines these questions by 


putting them into the context of this study. 
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C, RESEARCH QUESTIONS 


Finding answers to a problem entails first defining the problem. Three direct 
questions defined the problem-at-hand by asking: 
e How closely did the F-22 Spiral 2 implementation of EVM follow the 
criteria outlined in ANSI/EJA-748 Earned Value Management System? 


° To what degree did the F-22 Spiral 2 implementation of EVM fulfill its 


role as a management control system for avionics software development? 


® To what extent did the F-22 Spiral 2 program management (Government 
and Contractor) use EVM products to manage avionics development 


efforts? 


Answering these questions assessed the F-22 program’s use of EVM in managing 
avionics software development from a current (Spiral 2) perspective. The section-to- 
follow describes the approaches taken to answer these questions and, more importantly, 


explains why this subject matter was chosen for academic research. 


D. METHODOLOGY 


This section begins with a brief explanation of why the F-22 Spiral 2 program was 
chosen as a representative case study of the interaction between DoD software 
development and the EVMS that measures its progress. As established in the previous 
section, three questions essentially frame the research problem; this section concludes by 


matching these same questions with their primary answering method.8 


Why choose the F-22 Spiral 2 program as a case study? The first and foremost 
reason: access. Utilizing the professional relationships and contacts resulting from prior 
experience at the F-22 SPO greatly enhanced the probability of both a successful research 


effort and the delivery of useful recommendations. Access eased not only asking the 


8 The primary distinction was made since, in reality, each answer more or less utilized 
a mix of methods, with the primary method answering the majority of issues relating to 
its respective question. 


18 


questions and collecting answers; it facilitated the project with the knowledge of whom to 


ask. 


Along the same lines, the first-hand knowledge and direct observations gained 
from prior experience at the F-22 SPO created a sense of familiarity within the context of 
a research project seeking an unknown answer. This fomented a synergistic effect 
between knowledge gained while attending the Naval Postgraduate School (NPS) and the 
knowledge already possessed of EVM standard operating procedures at the F-22 
program. Consequently, this allowed the research to move beyond basic concepts and 


into the world of practical application, even prior to the information gathering phase. 


Another reason that supported studying the F-22 program was the program’s 
pending universality. Simply stated, pending universality means that the F-22 program’s 
software development exhibits trends? that experts believe will become commonplace in 
future DoD acquisition efforts. As such, having studied the interactions between 
unprecedented software development and EVM, resulting universal concepts (i.e., 
concepts applying to all programs, regardless of function) from the study should at least 


partially transfer to upcoming programs. 


The final reason behind choosing the F-22 program proved less complicated: the 
simple desire to help. The lack of clear-cut answers to the research problems, even with 
previous experience with the program, fostered unease with respect to EVM 
implementation within software development programs. Only after the attainment of new 
academia-based knowledge while attending NPS (e.g., Acquisition, Cost Estimation, and 


Research courses) was it felt that a helpful answer could become perceptible. 


In summary, the F-22 Spiral 2 program was chosen as the program-of-study due 
to its relative ease of access, familiarity, pending universality, and a desire to help the 
program. The following paragraphs concern the applications to the research problems 


introduced in the earlier Research Questions section. 


9 Specifically, the Joint Strike Fighter and Future Combat System both exhibit similar 
degrees of software complexity resulting from the weapon systems’ increased 
dependency on software and integrated constructs. 
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Answering “How closely did the F-22 Spiral 2 implementation of EVM follow 
the criteria outlined in ANSI/EIA-748-A-1998 Earned Value Management System?” 
relied mainly on a careful assessment of how EVM implementation for Spiral 2 
supported the 32 ANSI/EIA criteria. The text Earned Value Project Management by 
Fleming and Koppelman provided a majority of the assistance through its detailed 
description of each of the criteria. Those descriptions were aligned with procedures in 
place within the F-22 software program, in turn identified by interviews, portions of the 
questionnaire, and collected management documents. The degree of alignment answered 


the question for each of the criteria. 


“To what degree did the F-22 Spiral 2 implementation of EVM fulfill its role as a 
management control system for avionics software development?” was answered 
primarily by data collection. Specifically, Cost Performance Reports (CPRs) directly 
related to Spiral 2 efforts were distilled into raw cost and schedule measurements. A 
complete EVM analysis was accomplished and compared to outputs from SPO and GAO 
reports, in addition to testimony from individuals involved. The comparison, in essence, 


answered the question. 


Lastly, the EVM Questionnaire answered the question “To what extent did F-22 
program management (Government and Contractor) use EVM products to manage 
avionics development efforts?” The questionnaire aimed to assess the perceived 
usefulness of EVM within a software development context. If a given respondent 
answered positively to that question, they were asked to rate the practical value of EVM 
with respect to their program management duties. Interviews supplemented the data from 
the questionnaire, figuratively filling in the questionnaire’s information gaps uncovered 


throughout the course of the research project. 


In summary, accomplishing this project involved conducting interviews with 
subject matter experts, both at the F-22 SPO and Lockheed Martin, to assess their 
thoughts on exactly how well the program followed the ANSI/EIA-748-A-1998 
guidelines. In addition, results from the EVM Questionnaire were reported, along with 
deduced conclusions. This questionnaire surveyed occupational specialties involved with 


software development efforts, and summarizes opinions and knowledge related to EVM 
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and ANSI/EIA-748-A-1998. Finally, data and report collection assessed the information 


provided by, among other sources, the avionics development’s EVMS. 


E. FRAMEWORK 


Chapter I, /ntroduction, served three main purposes. First, the chapter provided 
context by discussing the history of avionics development within the larger F-22 
development effort (see the Background section). Second, it revealed the paper’s purpose 
and why this study may prove significant to future DoD efforts. Finally, it established an 
academic framework by identifying the problems this paper seeks to answer and how it 


answered them (see the Research Questions and Methodology sections). 


Chapter II, Literature Review, provides an informed foundation by examining 
current bodies of work that discussed applicable topics. Since research efforts focused on 
both software development and EVM, relevant information included references to 
suitable texts containing foundational thinking associated with these two topics—to 
include a basic primer for EVM. Additionally, contemporary ideas regarding the 


interaction between EVM and software development were explored. 


Chapter III, F-22 Implementation of ANSI/EIA EVMS Criteria, highlights the 32 
ANSI criteria. Using those criteria—as explained in the Earned Value Project 
Management text—an assessment was made of EVM implementation within F-22 
software development. Each criterion was analyzed separately and a conclusion was 
reached regarding the degree of alignment between implementation and its intended 
purpose. 

Chapter IV, F-22 EVMS Environment, examines the recent and current 
environment of EVM as it pertains to F-22 software development. This chapter seeks to 
consolidate questionnaire responses, interviews, observations, data collected, and the 
authors’ experiences regarding how the F-22 program applies EVM to software 
development. Specifically, the chapter aims to provide insight into how the current 
avionics contract environment, avionics suppliers, and recent avionics programs 


themselves may or may not contribute to difficulties in EVMS implementation. 
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Chapter V, Conclusion, presents a condensed synopsis of this research project’s 
outcome, includes a brief discussion on limitations with respect to the research project, 
and makes final recommendations to the EVMS where necessary. According to this 
paper’s research, these recommendations (if required) should serve to strengthen the F-22 


avionics program’s EVM system. 


F. SUMMARY 


This concludes Chapter I, Introduction. The next chapter, Literature Review, 
introduces the reader to existing academic works associated with Software Development, 
and EVM, and the interaction between the two. Chapter II also includes further 


background on F-22 avionics software development. 
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Il. LITERATURE REVIEW 


A. PREFACE 


This chapter strives to take the research topic, An Analysis of Earned Value 
Management Implementation in the F-22 System Program Office’s Software 
Development, and provide the reader with an informed and expert-based framework using 
a diverse collection of reports, papers, data, and experience. Providing the framework 


itself entailed examining existing bodies of work that discuss the following: 


e Unique Aspects of Software Development 
e Lessons Learned Regarding the F-22 Program and Software Development 
e Current F-22 Software Development Strategy—Spiral Modernization 


A completed analysis of these three areas will not only set the stage for the remainder of 
the paper, it will also enable the reader to begin framing desired scenarios against the 


backdrop of these real-world challenges and processes. 


B. UNIQUE ASPECTS OF SOFTWARE DEVELOPMENT 


1. Software Development Challenges 


Today the F-22 exists as one platform amid an exploding population of DoD 
equipment now relying heavily on software to perform their respective missions. Even 
so, the F-22 remains a precursor to the major defense weapons systems of tomorrow 
through its use of complex, embedded software. Consider the following: according to a 
Defense Science Board’s Task Force on Defense Software report (2000), military aircraft 
dependency on software increased from approximately 10 percent functionality on the F- 
4 to 80 percent functionality on the F-22—equivalent to a 2 percent per year increase 
(1960-1995). Simply stated, software has become ubiquitous within DoD acquisitions, 


and today’s high tech machines of war, in fact, depend on it at unprecedented levels. 


For example, even DoD’s historically “dumb” weapons—items such as air-to- 


ground bombs and artillery rounds—rate modernization funding to equip them with 
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advanced systems that boost functionality, precision, and lethality. However, equipping 
such munitions with GPS systems and autonomous guidance technologies predictably 
increases their own reliance on software to perform their functions.'° As a result, a given 
increase in performance causes some commensurate increase in complexity and risk (i.e., 


the inherent trade-offs between performance, schedule, and cost). 


A 1999 study performed by the Standish Group (an organization that studies 
information technology investments) brought attention to this suspected trend. The 
study found 31 percent of commercial, DoD, and combined commercial-DoD software 
development efforts resulted in cancellation. In addition, the study reported the following 
software development statistics (GAO, 2004):1! 


° Cost overruns of 189 percent 
° Schedule delays of 222 percent 


e Delivery of 61 percent of originally specified features or functions 


In an attempt to find some root cause of these dismaying statistics, the Defense 
Science Board, once again, investigated the area of software development. The Board 
found that software-intensive “programs lacked a well thought-out, disciplined program 
management and/or software development processes.” The findings went on to state that 
“meaningful cost, schedule, and requirements baselines were lacking, making it virtually 


impossible to track progress against them” (2000). 


Exploring this concept further involves analyzing the key differences between 
hardware and software development. For instance, when compared to hardware, software 
tends to propagate change effects at a higher rate. Furthermore, software exists more in 


the intangible realm of data and logic, versus physical components. Finally, software has 


10 Examples are the US Air Force Small Diameter Bomb (SDB) and US Army 
Excalibur weapons. The SDB program is a 250-Ib class guided munition currently under 
development at the USAF Air Armament Center (Picatinny Arsenal News Release, 
2005). The US Army’s Excalibur program is a howitzer-fired munition that uses GPS to 
guide (in flight) to within 10 meters of its target (Ruscetta, 2005). 


11 Percentages were based on comparison with initial baseline. 
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limited standardized design methods, components or structure when matched up against 


hardware. 


These differences mean that a typical software development project 
underestimates the development schedule when planners employ methods used to predict 
non-software development. It appears the very nature of software causes development 
issues that translate into cost, schedule, and performance concerns. Therefore, while 
labeling a development effort as “hardware” certainly does not grant immunity from 
these challenges and critiques, the complexity and prevalence of embedded software in 


weapons systems heightens the probability that problems do occur. 


Software’s nature does not completely differentiate itself from hardware, 
however. Software, in many cases, requires full integration with hardware. As such, it 
must share at least some fundamental characteristics with hardware to facilitate said 
integration. The Defense Acquisition University (DAU) stated that some of the key 
similarities between hardware and software include: functional decomposition; 
traceability to system requirements; accountability by task; progress monitoring; and 


reliance on operating principles and constraints (2006). 


As for DoD specific studies on the matter, the Acquisition, Technology, and 
Logistics (AT&L) Knowledge Sharing System (AKSS) summarized the typical problems 
defense acquisition programs have encountered over the years. For one, the dynamic and 
rapidly evolving nature of software development makes it difficult to adhere to an 
original baseline. Program managers and other decision makers lack basic software 


knowledge, which only aggravates the problem of baseline adherence. 


Similarly, end-product-users typically cannot accurately convey requirements, 
and promulgate requirements creep!2 throughout the development phase (once again 
hampering the baseline). This problem in turn leads to joint software and hardware 


development either starting or becoming uncoordinated, either directly because of poor or 


12 “A tendency for product or project requirements to increase auring development 
beyond those originally foreseen, leading to features that weren't originally planned and 
resulting risk to product quality or schedule” (Johnson, 2005). 
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non-existent software development metrics or as a result of inadequate software testing 


programs. 


AKSS provides a final assertion that effectively creates a foundational 
predicament with respect to the previously listed problems. DoD personnel generally 
(with few exceptions) lack fundamental software development knowledge: that dearth of 
knowledge will continue into the foreseeable future until DoD can effectively compete 


with the private industry for software engineers. 


This segment contained a brief discussion of software development challenges; 
the next topic concerns the distinctive software development lifecycle, and uses this 


context to further compare and contrast software with hardware. 


2. Software’s Lifecycles 


Fundamental characteristics aside, other differences pertinent to this paper exist 
between hardware and software development. Professional organizations such as the 
Carnegie Mellon Software Engineering Institute (SEI) and the USAF Software 
Technology Support Center (STSC) recognized this and bestowed a unique lifecycle 
upon software development separate from hardware development. Although different 
than the lifecycle phases typically imparted to classic hardware-intensive efforts, note the 
similarities and parallels between the phases of the software lifecycle and those of 
hardware (2005; 2003): 

e Requirements — Through interfacing with the customer, the developer 
analyzes operational problems or needs and translates them into functional 
requirements. This Systems Engineering process results in lower-level, 
detailed functional requirements traceable to higher-level requirements. 
Contrast to Concept Refinement and Technology Development within the 


Acquisition Framework (Figure 4). 


° Design — This phase involves definition of the software structure. It 
analyzes specific solutions and approaches and chooses the best 


alternative based on cost, schedule and performance parameters. Two 
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design reviews within this phase typically approve the Preliminary Design 
(the initial software architecture) and the Detailed Design (functional 
modules and interfaces). Contrast to Technology Development and System 
Development and Demonstration within the Acquisition Framework 


(Figure 4). 


Implementation (Development) — This phase involves actual coding of 
software. Coding usually entails an iterative approach consisting of 
subsystem (component) unit development and testing prior to integration 
testing within the main software build. Results from that testing in turn 
help develop yet another round of coding. Contrast to System 
Development and Demonstration within the Acquisition Framework 


(Figure 4). 


Testing — This phase typically involves three types of testing: Unit 
Testing, Integration Testing, and Acceptance Testing. As discussed 
above, accomplishing the first two types entails many cyclical trials prior 
to proceeding with acceptance testing, which verifies performance against 
requirements. Contrast to System Development and Demonstration within 


the Acquisition Framework (Figure 4). 


Deployment — Anticipate this phase to field the software product in its 
intended environment. Also, users become familiar with the system via 
training. Once complete, it finalizes the system development effort. 
Contrast to Production and Deployment and Operations and Support 


within the Acquisition Framework (Figure 4). 


Maintenance — Depending on the need for enhancement, fixes, or 
modifications, this phase ranges in scope from a minimal to a Herculean 
effort even larger than the original development. This phase typically 
costs far more than the original development effort. Changes in software 
this late in development come with a heftier price tag than the same effort 


undertaken during an earlier phase. Contrast to Production and 
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Deployment and Operations and Support within the Acquisition 


Framework (Figure 4). 
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As alluded to within the previous section, one of the inherent challenges with 
software development efforts lies in premature migration into the implementation phase 
prior to sufficient completion of the requirements and design phases. In an assessment of 
commercial software development companies, the GAO identified best practices that 
included the need for management to protect against missing, vague or changing 
requirements that negatively impact programs. Using commercial industry as the 
standard, GAO identified a benchmark of setting 95 percent of requirements by the end 
of the requirements phase, and 98 percent by the end of the design phase (GAO, 2004). 
For the military, the need for fully defined and stable requirements presents a unique 
challenge. In its report to the Secretary of Defense in 2006, the Defense Acquisition 
Performance Assessment panel stated the following: 

The [DoD] Acquisition System must deal with external instability, a 

changing security environment and challenging national security issues. 

The Department must be agile—to an unprecedented degree—to respond 


quickly to urgent operational needs from across the entire spectrum of 
potential conflicts. (p. 7) 


Balancing this need for flexibility against the recognized need for requirements stability 
in software development heightens the need for flexible, disciplined program controls 


within well-managed programs. 


Defining and understanding the different phases of the software lifecycle is only 
the first step to successfully managing a software program. As with any project planning 
effort, managers must identify those critical factors necessary to determine a software 
program’s success or failure. The five critical factors identified by the Air Force’s STSC 
are: Quality, Cost, Schedule, Performance and Supportability. For each of these factors, 
the project manager must develop appropriate plans, criteria, expectations, measures and 
controls to ensure the program stays on course. Since its inception, EVMS has proven a 
powerful tool for measuring and controlling the factors of cost and schedule. Although it 
“requires a fully defined project up front and bottom-up cost estimates...it can provide 
accurate and reliable indication of cost performance as early as 15 percent into the 


project” (STSC, 2000). 
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This section defined and represented the difficulties implicit within a software 
development program. The next section discusses the performance measurement of 


software development within the context of the F-22 program. 


C. LESSONS LEARNED REGARDING THE F-22 PROGRAM AND 
SOFTWARE DEVELOPMENT 


Current research pertaining to measuring the progress of software development 
supports the assertion that, in the initial stages of development, software efforts track 
much like their hardware counterparts. In other words, it is relatively easy to apply EVM 
to the first two phases—Requirements and Design. However, this assumes that the 


program in question has adequate cost and schedule controls. 


Two recent assessments of the F-22 program provided recommended changes 
regarding the interactions between its software development and its EVMS. Between the 
assessments exists a common theme: the program needs better cost and schedule 
controls. The first report was written by the Government Accountability Office (GAO); 
the RAND Corporation generated the second report. 


In its 2004 report, DEFENSE ACQUISITIONS: Stronger Management Practices 
Are Needed to Improve DoD’s Software-Intensive Weapon Acquisitions, the GAO 
recommended DoD require its software-intensive development contractors to first collect 
and regularly report metrics related to software cost, schedule, size, requirements, tests, 
defects and quality. In its next recommendation, the GAO suggests that DoD, in 
cooperative effort with its contractors, develop “‘an earned value management system that 
reports cost and schedule information at a level of work that provides information 


specific to software development.” 


More recently, in 2005 Younossi et al. identified in Lessons Learned from the 
F/A—22 and F/A—18E/F Development Programs the need to have EVM data “monitor and 
manage program costs at the level of integrated product teams.”” However, merely stating 
these controls should be put in place doesn’t necessarily equate to examination and 


understanding of all implications related to the complexities of applying EVM. This is 
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especially evident when considering the complex, dynamic, and unique characteristics of 


software development. 


Avionics remains a critical and arguably the most-complicated system of the F- 
22. It heavily impacts both cost and schedule, and has done so over a significant period 
of the program. Both program and contractor officials admitted that the program failed to 
follow their stated software strategy: to collect metrics and manage to those metrics. 
This failure facilitated the loss of program cost and schedule control. Further 
investigation revealed that other cost and schedule pressures within the F-22 program 
contributed to a failure of desired software metrics. These pressures kept the program 
from providing its managers the necessary metrics for sufficient oversight of the overall 


progress of software development efforts (GAO, 2004). 


D. CURRENT F-22 SOFTWARE DEVELOPMENT STRATEGY —- SPIRAL 
MODERNIZATION 


As discussed in the previous chapter, the F-22 program has undergone significant 
external pressures requiring adaptation to changing strategic and tactical threats, even 
while fighting for funding from shrinking DoD budgets. All this while developing one of 
the most technically complex systems fielded to date by DoD. These schedule pressures, 
changing requirements, technical risks, and funding instabilities haunted the F-22 EMD 
program through its conclusion in 2005. With the need to deliver unmet EMD 
requirements and modernize the fighter to meet emerging threats, Air Force leadership 


initiated a follow-on effort to EMD in 2003. 


Unlike its predecessor, the F-22 Modernization Program was not contracted under 
one behemoth contract. Rather, it was contracted with Lockheed Martin Aeronautics 
(LM Aero) under an Indefinite Delivery Indefinite Quantity (IDIQ)!3 contract titled the 
Raptor Enhancement Development & Integration (REDI) Contract. The REDI contract, 


modeled after a highly successful C-17 modernization contract, would serve as the single 





13 The Federal Acquisition Regulation (FAR) defines an IDIQ as “a contract for 
supplies that does not procure or py a firm quantity of supplies (other than a 
minimum or maximum quantity) and that provides for the issuance of orders for the 
delivery of supplies during the period of the contract” (Subpart 16.501-1). 
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contract for “planning, analysis, design, development, qualification, test and 
documentation of performance enhancements” necessary for the F-22 mission (F-22 
System Program Office, 2003). Once awarded, the basic IDIQ contract would authorize 
work via individual delivery orders focused on specific tasks or development efforts. The 
magnitude of the delivery orders varied, ranging in value from several hundred thousand 


dollars to several hundred million dollars. 


The first delivery order (DO 0001) awarded under the new REDI contract was the 
System Engineering/Program Management effort. This DO was the starting point for any 
enhancement considered for the F-22 and charged LM Aero “to provide overall Systems 
Engineering and Program Management in support of the F/A-22 program to maintain 
effective incorporation of changes into the weapon system” (F-22 System Program 
Office, 2003). It also served as the overarching architecture, accomplishing all early and 
up-front analysis on an enhancement candidate before committing additional resources 
and formally proceeding with a stand-alone delivery order. Therefore, DO 0001 is where 


Spiral 2 found its start. 


In 2002, when the F-22 program was still three years from completing EMD, its 
users began looking forward to what the fighter jet would look like when it was declared 
mission ready at IOC; the users realized it would fall short in some capabilities. As these 
capabilities were identified, quantified, and prioritized, a list began to emerge that would 
define the first upgrade to the jet one year after IOC declaration in December 2005. The 
upgrades focused on a software-only evolution to the avionics Operational Flight 
Program. Although the upgrade was later identified by LM Aero in more accurate terms 
as Block 20, the government organizations and documentation continued to call the 
program by the original name, a name that captured one of the latest buzz words in DoD 


acquisition—Spiral 2.14 


14 Spiral 2 is not a spiral product as the name would imply. While each spiral of the 
F-22 modernization program built on the preceding spirals capabilities, the upgrades 
more closely resembled increments versus spiral releases. In 2004 considerable 
discussion between the F-22 program and senior Air Force leadership centered around the 
correct terminology for the modernization upgrades. In the end it was decided to 
continue to call them spirals while they would be managed internally by government and 
contractor personnel as block upgrades. 
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With EMD and the IOC baseline representing Spiral 1, Spiral 2 represented the 
first of several upgrades planned for the jet in the modernization program. As the 
pathfinder, Spiral 2 established the procedures and template for the much larger and more 
complex software/hardware upgrades of Spirals 3 and 4 that would follow. In March 
2003, within the scope of DO 0001, requirements analysis for Spiral 2 started creating a 
list of potential enhancements called candidates. These candidates were further 
developed and carried forward to the end of this phase based on several constraints: user 
priority, available funding, and schedule alignment. As stated above, Spiral 2 was a 
schedule driven upgrade that planned to deliver software-only upgrades to the OFP not- 
later-than one year after I[OC—or December 2006. These constraints—along with 
funding limitations—would eventually narrow the list to a handful of approved 
candidates to carry forward to the follow-on delivery order. Spiral 2 completed 


requirements analysis in May 2004 and was ready for the next phase of the program. 


Not only was the modernization contract broken into individual contract vehicles 
called delivery orders, but the larger spiral upgrades were broken into different delivery 
orders. The plan developed for the spirals accomplished requirements analysis on DO 
0001. Next a separate delivery order was awarded for the detailed design of the upgrade. 
Upon completion of the design effort, another delivery order would be initiated to 
accomplish the coding, integrating, developmental testing and post-operational test 
updates to the upgrade. This would lead to a modification to the production contract that 
would field the completed product. Although this process resulted in tremendous 
pressures on the business processes in the program, it afforded the program the flexibility 
to adapt to the pressures that caused EMD to flounder so many times. As funding 
realities changed, as technology challenges were realized, and as requirement priorities 
shuffled, each transition between delivery orders enabled “on-ramps” and “off-ramps” for 


capabilities. 


For Spiral 2, the second contract effort was DO 0002. This contract was started in 
March 2004 and included all tasks necessary to accomplish Preliminary Design Review 
and Critical Design Review for the candidates identified in the contract. This was the 
first stand-alone Spiral 2 delivery order and utilized EVM as a management tool for the 
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duration of the contract. Unfortunately, the fluidity of requirements and funding 
combined with the lengthy timelines associated with awarding REDI delivery orders (up 
to ten months from solicitation to award), forced program management involved to award 
DO 0002 as an Undefinitized Contract Action (UCA). In Chapter IV, F-22 EVMS 
Environment, the impacts of UCAs on a program’s EVM will be discussed; prior to that, 


however, a short background on the use of UCAs seems appropriate. 


Lack of funding predictability, emerging technical requirements from the user, 
and lengthy business review processes were just a few of the challenges facing each 
incremental upgrade for the F-22. These challenges, combined with competition for 
business resources, often led to the initiation of efforts using UCAs. A UCA permits the 


initiation of an effort without a firm (definitized) contract in place.15 


While this paper does not try to tackle the complex issues surrounding the pros 
and cons of using UCAs, the risks associated with proceeding under undefinitized 
contracts makes this method a contracting tool limited to those instances where it is 
absolutely necessary. Although used by exception, highly scrutinized, and not typically 
desirable, in the F-22 program, use of this method of contract award has become 


prevalent. 


Even with the inherent speed of a UCA, Spiral 2 DO 0002 was completed in 
February 2005—two months past the original period of performance. The completion of 
the detailed design cleared the way for initiation of the software coding and integration 
phase. This effort was awarded in January 2005 under a partial UCA for DO 0019 on the 
REDI contract. (The remainder of the effort was authorized under a UCA in March 
2005.) As with DO 0002, this effort would proceed for a long duration under a UCA. 
Unlike DO 0002, it would be definitized (negotiated via a firm contract) prior to its 
completion. DO 0019 was definitized in October 2005 and is expected to complete in 
September 2007. 


15 The Defense Federal Acquisition Regulations Supplement (DFARS) defines an 
UCA as “any contract action for which the contract terms, specifications, or price are not 
agreed upon before performance is begun under the action” (Subpart 217.7401d). 
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E. SUMMARY 


This concludes Chapter II, Literature Review. The next chapter, F-22 
Implementation of ANSI/EIA EVMS Criteria, uses the 32 ANSI criteria to analyze each 
criterion separately and derive a conclusion regarding the degree of alignment between F- 


22 program implementation and the criterion’s intended purpose. 
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HI. F-22 IMPLEMENTATION OF ANSI/EIA EVMS CRITERIA 


A. PREFACE 


The 32 criteria identified in ANSI/EIA 748-1998 are recognized by both industry 
and DoD as the minimum standards for establishing a useful, functioning EVMS. 
Meeting these criteria is critical if an organization is going to be able to successfully use 
the EVMS as a management control tool. In order to assess the implementation of the 
EVMS in the F-22 program, these criteria were used to evaluate implementation of the 
EVMS in the Spiral 2 modernization program. This assessment was not intended to 
address whether LM Aero EVMS policy and procedures were sufficient. LM Aero has 
already demonstrated to DoD their processes comply with the criteria via their 
certification from industry. Instead, this was a subjective, qualitative assessment, based 
on the criteria objectives outlined by ANSI/EIA, data and procedures gathered from the 
F-22 program and the authors’ direct observations, which looked at how the 32 criteria 


were actually applied through the implementation of Spiral 2 contracts. 


For each criterion an assessment was made resulting in one of five ratings: 
excellent, satisfactory, marginal, insufficient and inconclusive. Based on supporting data 
that exceeded the purpose of its respective ANSI criteria, the following seven criteria 
were conferred an excellent rating: 4, 14-16, 19, 30, and 31. Adequate supporting data 
that met the purpose of its respective ANSI criteria, led to the following 15 criteria being 
conferred a satisfactory rating: 1-3, 5, 7, 10, 11, 13, 17, 22-25, 28, and 32. Due to 
supporting data that contained minor material failures related to meeting the purpose of 
its respective ANSI criteria, the following five criteria were conferred a marginal rating: 
6, 8, 12, 18, and 27. One criterion, 26, was conferred an insufficient rating based on 
supporting data that contained more than minor material failures regarding its respective 
ANSI criteria, the following criterion. Finally, an inadequate amount of supporting data 
for a given ANSI criteria, led to the following four criteria being conferred an 


inconclusive rating: 9, 20, 21, and 29. 
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The next section contains a more detailed discussion of these assessments for each 
of the criteria. Specifically, each criterion includes a brief overview of the intent of the 
criterion (Fleming and Koppelman, 2000), F-22 (LM Aero) policy/procedures for 
meeting the criteria (Lockheed Martin Aeronautics Corporation, 2005) and a discussion 


of the authors’ assessment for each criterion. 


B. CRITERIA ASSESSMENTS 


1. Group 1: Organization Criteria 


Table 2 summarizes the assessments provided to each of the five ANSI/EIA 
EVMS Organization Criteria. 


Criteria Assessment 


Define authorized work (WBS elements) 
Identify organizational responsibilities 


Integrate the system 
Identify overhead management Excellent 
Provide for performance measurement 
Table 2. Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Organization 
Criteria 





a. EVM Criterion #1: Define the authorized work elements of the 
program. A WBS, tailored for effective internal management 
control, is commonly used in this process. 


This initial criterion addresses the necessity of starting a program only 
after fully defining its requisite efforts. As hinted at within the criterion itself, military 
acquisition programs require a program-specific WBS which, through its very nature, 
satisfies this criterion.!6 Consequently, work requested by a customer not identifiable 
within an already developed WBS should be considered out-of-scope, and the customer 
and/or contracted organization should seek FAR-approved authorization methods to 


begin such work (in this case, the Air Force and LM Aero, respectively). 


16 As described per MIL Handbook 881. 
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With respect to Spiral 2-related work, LM Aero stated that tasks were 
authorized for accomplishment through modifications to the basic Raptor Enhancement 
Development and Integration (REDI) contract (via the F-22 SPO’s contracting officer). 
Once signed, these Delivery Orders (DOs) then energized LM Aero’s Business 
Management division to initiate a Sales Order, which by itself serves as notification of 
task authorization to Spiral 2’s management team at Lockheed. The management team 
analyzed the DOs, Sales Orders, and Statements of Work (SOW) within proposals for the 
actual allocation of authorized tasks to their appropriate WBS elements. At the direction 
of ASC/YFK (the customer contracting officer) via a Contract Data Requirements List 


(CDRL), an official WBS was drafted. 


LM Aero communicated that the WBS was structured to conform to the 
latest version of MIL Handbook 881 and that the WBS includes all Contract Work 
Breakdown Structure (CWBS) elements specified for external reporting by the CDRL. 
Direct observation of the Cost Performance Reports (CPRs) supports this assertion. The 
WBS appeared to capture all authorized work due to the lack of changes to the WBS 
throughout the lifecycle of the Spiral 2 program. In conclusion, with respect to LM 
Aero’s actions concerning the definition of authorized work for the Spiral 2 program, by 


meeting this criterion to the letter this EVMS program warrants a satisfactory rating. 


b. EVM Criterion #2: Identify the program organizational 
structure, including the major subcontractors responsible for 
accomplishing the authorized work, and define the 
organizational elements in which work will be planned and 
controlled. 

The Organizational Breakdown Structure (OBS) must be established to 

ensure that all elements of the WBS (established per Criterion #1), are assigned to a 
specific organization or individual. The establishment of the relationship between the 
OBS and WBS results in an OBS/WBS assignment matrix. This product ensures clearly 


defined responsibility for each task’s completion. 


LM Aero establishes their OBS according to an Integrated Product Team 


(IPT) structure. Here each level or tier has IPTs respectively assigned to one IPT above 
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them, ensuring that any IPT will only be subordinate to one IPT (Figure 5 provides an 
example of the IPT breakout for Spiral 2). This approach enables one element of the 
WBS to be assigned to one IPT (or major subcontractor). The interface between one 
organization and one WBS element is what defines a cost/schedule account. While each 
WBS element may only be assigned to one organization, multiple WBS elements may be 


assigned to any one organization. 


F-22 DO0002 IPT Structure 


Air Vehicle 
Air Vehicle Systems 
Engineering and Integration 
Team (SEIT) 
Air Vehicle Systems 
Build Team 
Mission System & Software 
Avionics SEIT 
Core Processing 
Communications, Navigation 
& Identification! 
Stores Management System 
Display Products 
Electronic Warfare 
Mission Avionics Software 
Radar 
Sustainment & Modifications 


Modifications and Heavy 
Maintenance 
Support Equipment 
Support Data 
Support Services 
Training 
Weapon System Integration & 
Technology 
Flight Test 
Flight Termination System 
Test Plan & Support 
System Integration 
Program Operations 
Modernization 


Figure 5. | Spiral 2 IPT Breakout 
(From: Spiral 2 CPR (Format 2), 2004) 





40 


LM Aero appropriately established a detailed OBS/WBS assignment 
matrix for all Spiral 2 contract efforts. Although the initial delivery order for 
Requirements Analysis Phase did not establish this product in as much detail as the two 
follow-on efforts, it did provide the necessary relationship between the WBS elements 
and organizations assigned to each of these elements. As discussed above, this effort was 
accomplished as part of an overarching delivery order that encompassed several 
modernization efforts. As a result, the WBS elements for the Spiral 2 effort were at a 
higher level than those developed and assigned for the two Spiral 2 specific delivery 


orders. 


Additionally, a critical concern with this criterion that will be repeated for 
many other criteria is the timeliness of the establishment of the OBS/WBS assignment 
matrix. The Integrated Baseline Review (IBR)'’ for each of the three Spiral 2 efforts 
was held significantly later than the initiation of the contract effort.18 Based on the 
authors’ experience, this was driven by the lack of a definitized contract, changing 
government requirements and immature modernization processes. The LM Aero cost 
accounting process made the creation of cost/schedule accounts mandatory in order for 
work to proceed, without the convening of an IBR. There are, however, significant 
questions regarding whether these cost/schedule accounts were established in a planned 
and controlled manner, a manner that ensured appropriate development of relationships 
between all WBS and OBS elements. These concerns notwithstanding, with the intent of 
this criterion being met by LM Aero actions and procedures warrants a satisfactory 


rating. 


17 An IBR is typically held within the early stages of a contract period of performance 
and it “establishes a mutual understanding of the project performance measurement 
baseline” and provides “an agreement on a plan of action to evaluate the risks inherent in 
the program measurement baseline and the management processes that operate during 
project execution.” (DAG, 4.3.2.4.2) 


18 See Chapter IV, Section B for a discussion on delayed Spiral 2 IBRs. 
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Cc. EVM Criterion #3: Provide for the integration of the company’s 
planning, scheduling, budgeting, work authorization, and cost 
accumulation processes with each other, and, as appropriate, the 
program WBS and the program organizational structure. 


To ensure project goals are given priority over any one functional area’s 
goals, the program must employ an integrated, single management control system using 
common information from the programs functional areas. The integration of master 
scheduling, cost estimating, work authorizations, budgeting and cost accumulation must 
work within a single database to ensure managers can get a complete picture of program 


health and make management decisions based on inputs from all functional disciplines. 


The LM Aero defines an integrated process for developing project 
schedules and budgets, authorizing the work associated with those plans and 
accumulating and reporting costs and schedule progress consistent with the established 
WBS elements of the contract. During the development of the project schedule, activities 
are defined along with their interdependencies with other activities. These activities will 
become the basis for measuring performance in terms of resource requirements (i.e., cost 
to complete) and individual work package progress. Similarly, budgets are developed, 
authorized and accumulated by individual WBS elements (cost/schedule accounts), 
allowing measurement of program health at both the discreet work package level and 
“rolled up” higher WBS levels. This “rolling up” of program cost is accomplished 
mechanically and allows for flexible reporting of program status at varying project levels 


based on the desired focus. 


The integration of the different functional areas of the Spiral 2 WBS and 
OBS was consistently observed. The development of the detailed Integrated Master 
Schedule (IMS), authorization of budget, and cost accumulation were all clearly tied to 
the program WBS. Additionally, organizational responsibility was clearly defined via the 


assignment of cost/schedule accounts to individual IPTs. 


One concern, that will be addressed later, is the relationship between the 
project IMS and the detailed IPT activity schedules that supported the resource-loaded 
IMS. Specifically, the issue was how well they were linked and controlled. IMS and 
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associated cost/schedule accounts were resource loaded and managed per defined work 
authorizations and performance measures. Lower tier IPT schedules developed to 
support the assigned activities were not directly linked to the IMS and, therefore, 
permitted IPTs to “interpret” their detailed schedules and take credit for progress against 
the IMS tasks. This approach is not consistent with the intent of this criterion which 
asserts the goal of measuring all progress against project goals versus individual IPT 
goals. This concern does not affect the intent of this criterion enough to warrant a 


negative assessment; therefore a satisfactory rating was given to this criterion. 


d. EVM Criterion #4: Identify the company organization or 
function responsible for controlling overhead (indirect costs). 


Adequate identification, allocation and tracking of program indirect costs 
is a concern for many programs. This is the first of four of the thirty-two EVM criteria 
that deal with management of indirect costs (others are Criteria #13, #19, and #24). 
Although not directly controlled by any individual project manager, indirect costs must 
still be clearly identified as a category, formally documented, and assigned to individual 


managers responsible for authorization and control. 


Control of overhead rates and application to a specific LM Aero contract 
is the responsibility of the Overhead Section of the Aeronautics Controller. Surveillance 
of overhead costs allocated to the contract, however, is the responsibility of the project 


and functional managers. 


Overhead rates applied to Spiral 2 contracts were used by LM Aero based 
on rates negotiated with Defense Contract Management Agency (DCMA). DCMA was 
also responsible to review performance reports to ensure rates were being applied in 
accordance with agreements. Although negotiation of each Spiral 2 contract focused on 
the applicability and appropriateness of applied overhead categories, once the negotiated 
indirect costs were authorized per the negotiated contract, the only control measure for 
the LM Aero and government program managers was a comparison of allocated overhead 


to the overhead portion of contract budget. LM Aero procedures and performance 
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regarding the management of overhead indicated an excellent rating was appropriate for 


this criterion. 


e. EVM Criterion #5: Provide for integration of the program WBS 
and the program organizational structure in a manner that 
permits cost and schedule performance measurement by elements 
of either, or both, structures as needed. 


In order to measure performance, a standard must exist. This criterion 
concerns the formation of that standard, known within the military acquisition 
community as the program baseline. This concept has proven so important that three 
other criteria will deal with the issue of implementing a baseline.!9 This criterion 


concerns only the foundations of that baseline—that is, the integration of WBS and OBS. 


According to LM Aero, management used IPTs to integrate Spiral 2’s 
WBS and organizational structures. In general, IPTs had the responsibility of 
accomplishing tasks within specific CWBS elements. LM Aero allowed that, in general, 
an IPT may have more than one assigned CWBS element, but that IPTs did not share a 
single CWBS element. This assignment of a specific CWBS element to an IPT 
established the cost/schedule account, the base level of control which enabled future cost 


and schedule measurements. 


The proof of these assertions lay in the fact that BCWS, BCWP, and 
ACWP were available at the cost/schedule account level. This data also was directly 
summarized to only one higher CWBS and only one IPT structure element in the CPR. 
Although the data was theoretically available for summarizing at any structure level, 
reports were only generated at the detail that LM Aero was contractually obligated to 
transmit. In the case of Spiral 2, the CDRL called for three reportable tiers within the 
CPR. In conclusion, with respect to LM Aero’s actions concerning the integration of 
program WBS and OBS for the Spiral 2 program, the EVMS program warrants a 


satisfactory rating for meeting the letter of this criterion. 


19 ANSI/EIA EVMS Criteria #17, #18, and #25. 
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2. Group 2: Planning, Scheduling and Budgeting Criteria 


Table 3 lists the assessments for each of the ten Planning, Scheduling and 


Budgeting Criteria in Group 2 of the ANSI/EIA EVMS Criteria. 


Criteria Assessment 


Schedule the work 

Identify products, milestones and indicators 

Plan the Performance Measurement Baseline (PMB) 

Establish budgets for work Inconclusive 

Identify work packages 

Summarize work package budgets to control accounts 

Identify and control level of effort 

Establish overhead budgets 

Identify management reserves and undistributed budget. Excellent 

Summarize budgets to target cost Excellent 
Table 3. | Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Planning, 

Scheduling and Budgeting Criteria 





a. EVM Criterion #6: Schedule the authorized work in a manner 
that describes the sequence of work and identifies the significant 
task interdependencies required to meet the requirements of the 
program. 


When developing the IMS, a contractor must ensure that all activities 
required to complete the effort and the relationships between those activities are well 
defined. Many programs do not take the necessary steps to develop the IMS to the 
necessary level of detail. In order for EVM to be a useful management control tool, the 
IMS must be accurate with respect to allocation, consistency and traceability of budgeted 


schedule and resources. 


For LM Aero the process of developing a project schedule begins with the 
Contract Delivery Schedule and ends with fully defined activity schedules. The factors 
affecting the scheduling process include required resources, available resources, span 
times, activity relationships and external constraints. As these factors are considered, 
activities are defined that produce necessary interim and final product(s) of the contract 


effort. As activities are defined and assigned to work packages (which are in turn 
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assigned to a given cost/schedule account), schedule requirements are identified for each 
activity and resulting start/completion dates can be identified for a particular work 
package merely by identifying the start date for its earliest activity and the completion 
date for its final activity. The LM Aero process also highlights the critical and sometimes 
overlooked step of ensuring that interdependencies are clearly defined and understood for 


project work packages and their activities. 


This is one of the criteria significantly impacted by the F-22 culture of 
changing requirements. While LM Aero took steps to build and manage to detailed and 
linked schedules, the frequent impacts of changing requirements made developing a 
baseline schedule very difficult. This culture of change resulted in Contract Delivery 
Schedules that did not provide confidence for the IPTs charged with developing the 
detailed work package activity schedules. All three Spiral 2 contracts were initiated and 
progressed several months (up to 50 percent of the contract period) prior to the 
establishment of a baselined IMS. This was especially troubling when considering the 
fact that exit criteria for the first two contracts included detailed IMS for their respective 


follow-on contracts. 


The program recognized the need to have a detailed schedule prior to the 
initiation of a contract; however, the fluid requirements (as well as modernization 
program immaturity) made this unattainable for all three Spiral 2 contracts. As discussed 
above, once the detailed IMS was defined, it was noted in many cases there weren’t 
mechanical linkages between the IMS activities and the detailed activity schedules used 
by the IPTs. The approach used by LM Aero in the case of Spiral 2 was to define IMS 
activities by time span versus the detailed IPT tasks that would actually be required to be 
performed for completion of the respective activity. These shortcomings led to a 


marginal rating for this criterion associated with scheduling the work. 
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b. EVM Criterion #7: Identify physical products, milestones, 
technical performance goals, or other indicators that will be used 
to measure progress. 


In order to take credit for earned value, a project must first identify the 
meaning of value. This criterion requires the contractor identify tangible measures for 
determining how much value has been earned in the progress of the effort. In the case of 
software development, where many interim products are difficult to measure, this is one 


of the more challenging criteria to meet. 


LM Aero policy is lightly defined for this criterion. It does identify the 
requirement to objectively measure progress based on completion of tangible products, 
but it does not provide guidance on how to determine what is tangible. Their policy also 
states that “in most cases” progress will not be reflected for a particular activity until the 
activity has been assessed as complete (Lockheed Martin Aeronautics Corporation, 


2005). 


This criterion is one in which the authors have seen government customers 
lose confidence in contractor EVMS. The challenge is for the contractor and government 
to agree on the value of the products identified as program measures. Additionally, one 
could argue not all tangible products are necessarily measurable in terms of value— 
particularly in the case of software development. For example, when a lower tier IPT 
finishes coding a software product, how much value should be assigned to this product 


before it has been fully tested in an integrated fashion? 


While many interfaces can be tested, most of the problems in software 
testing occur during integration testing versus unit testing. To give only minimal value to 
the product prior to integrated testing might undervalue the product and not accurately 
reflect the progress of the project. However, too many times a program takes too much 
credit for completion of the unit and later shows unfavorable variances when integrated 


testing identifies unplanned rework for the product. 


Recognizing these challenges in Spiral 2, LM Aero attempted to assign 
values to the completed interim products based on the possibility (likelihood!) of 


problems with integrated testing that would require rework. The determination of how 
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much value to assign to these interim products and how much budget to assign to 
integration test and rework activities was a process that required past experience. In this 
area LM Aero and their major supplier, Boeing, both had tremendous experience 
developing fighter aircraft; however, the new challenges of developing the first fully 
integrated avionics system combined with the years of software challenges seen in F-22 
EMD, made this criterion one that required special focus from the government. LM Aero 
efforts to circumvent all of these challenges and adhere to their well-defined procedures 
in this area of identifying products, milestones and indicators yielded a satisfactory rating 


for this criterion. 


Cc. EVM Criterion #8: Establish and maintain a time-phased budget 
baseline at the control account level, against which program 
performance can be measured. Initial budgets established for 
performance will be based on either internal management goals 
or the external customer-negotiated target cost, including 
estimates for authorized but undefined work. Budget for far- 
term efforts may be held in higher-level accounts until an 
appropriate time for allocation at the control-account level. On 
government contracts, if an over-target baseline is used for 
performance measurement reporting purposes, prior notification 
must be provided to the customer. 

The “time-phased budget baseline” mentioned above describes the distinct 

EVM term known as Program Management Baseline (PMB).20 The PMB must include 
all authorized work, and thus becomes beholden to the success of most of the previous 
criteria. Unless a given program has a well-defined PMB with effective management 
control systems in place, that program has little to no chance of providing useful insight 


regarding performance status using EVM data. 


According to LM Aero, the PMB was established and maintained at the 
cost/schedule account level. For Spiral 2 the budget was based on the negotiated target 
cost, to include estimates for any contractually authorized but not negotiated changes 


(e.g., Undistributed Budget). The section of this criterion concerning reporting PMBs 


20 See discussion within the EVM Primer section of Chapter I. 
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reflecting an overrun (the “over-target baseline”) was rendered non applicable due to the 
fact that Spiral 2 experienced an underrun. However, LM Aero communicated that 
before any of their projects can implement an over-target baseline for PMB purposes, the 
LM Aero F-22 program manager and Director of Cost Management Integration must 


justify and provide prior notification to the customer—the F-22 SPO. 


Evaluating F-22 program implementation of this criterion necessitates 
dividing it into two parts: establishment and maintenance. The establishment half of this 
criterion warrants a satisfactory rating, given 1.) the establishment of the PMB within the 
Spiral 2 CPRs and Contract Funds Status Reports (CFSRs); 2.) the CPR and CFSR 
reconciliations with respect to authorized budgets; 3.) the CPR and CFSR reconciliations 
between the two reports themselves. The maintenance half of this criterion, much like 
Criterion #6, was significantly impacted by the F-22 culture of changing requirements. 
Specifically, late contract definitization (and subsequent IMS baselining) rendered the 


reported EVM data virtually useless. 


As also described in Criterion #6, the approach used by LM Aero in the 
case of Spiral 2 was to define IMS activities by time span versus the detailed IPT tasks 
actually required to be performed for completion of the respective activity. 
Consequently, the final CPR reflected a PMB-derived EAC of $46M even while noting a 
“most likely” EAC of $24 million. Thus, up to the point of the final CPR submission on 
January 2005, all EVM data reflected performance based on a benchmark 192 percent 
more than the yet-to-be definitized EAC. As a result, the maintenance half of this 
criterion warrants an insufficient rating. In conclusion, with respect to Spiral 2 
establishment and maintenance of a time-phased budget baseline at the control account 
level, the EVMS program warranted an overall marginal rating for meeting the purpose 


of this criterion. 


49 


d. EVM Criterion #9: Establish budgets for authorized work with 
identification of significant cost elements (labor, material, and so 
on) as needed for internal management and for control of 
subcontractors. 


Criterion #9 pertains to total project budgeting, which a program can only 
accomplish through a comprehensive list of cost elements. As the above alludes to, 
effective formal control systems must accompany the roll-up of cost elements. Budgeted 
values must equate to negotiated project costs, from the standpoint of both supply (sub- 


contractors) and demand (F-22 SPO). 


From LM Aero’s perspective, work packages and planning packages were 
budgeted by elements of cost. Specifically, discrete portions of the total contract budget 
base were allocated to each Cost/Schedule Account Manager (C/SAM) through the 
Budget Ledger. LM Aero states that subcontracted CWBS elements were identified 
within the accounting system by “unique work orders and work-in-process subaccounts” 
(Lockheed Martin Aeronautics Corporation, 2005). Within the EVMS, these accounting 
system data items were represented by specific cost element codes. Once again, the use 
of a shared CWBS by all elements of the project organization assures a common 
understanding, consistency for planning and performance, and effective oversight of all 


contractually authorized tasks. 


In practice, two issues surfaced regarding this criterion. First and 
foremost was a lack of verifiable supporting data. As explained within the preface of this 
chapter, the above paragraph describes policy as opposed to implementation. It should 
not, therefore, warrant consideration as proof positive of an effective execution of this 


criterion. 


The second area of concern stemmed from the description of the Budget 
Ledger. According to LM Aero’s own policy, the ledger may authorize the budget by 
CWBS element not only in terms of total dollars, but also in terms of less discrete cost 
elements such as labor hours/dollars and burden overhead dollars (to name a few 
examples). If the Budget Ledger described a C/SAM account's budget by cost element, 


responsibility laid with the IPT leader for translation of said cost element into total dollar 
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terms. The resultant unclear “flexibility to budget the work packages/planning packages 
of the cost/schedule account in whatever mix of resources deemed appropriate” was seen 
as a material weakness with respect to this criterion (Lockheed Martin Aeronautics 
Corporation, 2005). However, that concern also lacked specific, verifiable supporting 
data. As such, with respect to Spiral 2 establishment of budgets for authorized work as 
needed for internal management and for control of subcontractors and maintenance of a 
time-phased budget baseline at the control account level, the EVMS program warranted 


an inconclusive rating due to lack of supporting data. 


e. EVM Criterion #10: To the extent that it is practical to identify 
the authorized work in discrete work packages, establish budgets 
for this work in terms of dollars, hours, or other measurable 
units. Where the entire control account is not subdivided into 
work packages, identify the far-term effort in larger planning 
packages for budget and scheduling purposes. 


This criterion further expands on Criterion #8, establishment of a 
definitive PMB, by identifying the need for discreetly defined work packages. Although 
“far-term”’ is not defined, it is recognized that at some point it becomes non-value-added 
to attempt discrete definition of work packages that are too far removed from the current 
state of the program. This criterion also identifies the requirement to establish 
measurable metrics for assessing the amount of work accomplished at any point in the 


effort. 


LM Aero policy does not stipulate the delineation between near-term and 
far-term activities. It does, however, provide the planners more guidance than the 
ANSIJ/EIA criterion. While LM Aero directs that “all work [be] planned for the duration 
of the contract” (Lockheed Martin Aeronautics Corporation, 2005, p12), it also 
recognizes the lack of certainty or definition that may exist in activities planned outside 
of the current year of effort. The general approach is current year activities will be part 
of well-defined work packages while out year activities will be assigned to planning 


packages. It is important to note that while there exists less detail associated with the 
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planning packages, they, like the work packages, are still defined with planned 


start/finish dates, quantity (units, hours, etc.) and dollars of resource required. 


Due to the relatively short duration of all three Spiral 2 contract periods, 
LM Aero was able to develop discreetly defined work packages for the majority of the 
work performed on each effort. The first two contract efforts were each approximately 
one year in duration. The third was just over two years. The primary challenge of this 
criterion was the ability to define discreet work packages in an environment of changing 
requirements. With all three of these efforts proceeding under a UCA contract, there 
existed a lack of certainty regarding the content of the contract effort and contract 
funding. This led to more work than desired being held in either planning packages or 
undistributed budget (see Criterion #14). In conclusion, LM Aero implementation of this 
criterion, identifying the need for discreetly defined work packages, resulted in a 


satisfactory rating. 


f EVM Criterion #11: Provide that the sum of all work-package 
budgets, plus planning-package budgets within the control 
account, equals the control-account budget. 


The sum of all Spiral 2 work and planning-package budgets should have 
been equal to their respective control account budgets. Furthermore, each of the control 
account budgets must have been related to a specific SOW. The only account that was 
not included was the Management Reserve (MR) account, held in general outside the 


purview of the performance baseline. 


LM Aero acknowledged that distribution of Spiral 2-negotiated target cost 
was made from the individual cost/schedule accounts (aka control account) to their 
respective work and planning-packages. In all cases, LM Aero ensured that the sum of 
the budgets assigned to these packages equated to the total dollar budget authority of the 
cost/schedule account. Fulfillment of Criterion #1 made certain that each of the control 


account budgets related to a specific SOW. 


The CPRs generated in support of Spiral 2 reporting substantiated LM 


Aero’s policy regarding this criteria. An assessment of the data showed all work 
a2 


packages correctly summed up to their respective control accounts throughout the period 
of reporting (Apr 2004 through Jan 2005). MR grew during the reports, as expected 
given the at-the-time anticipated underrun. In conclusion, with respect to Spiral 2 
summing of all budgets within a control account equating to the authorized total of that 
control account, the EVMS program warranted a satisfactory rating for meeting the 


purpose of this criterion. 


g. EVM Criterion #12: Identify and control level-of-effort (LOE) 
activity by time-phased budgets established for this purpose. 
Only that effort that is unmeasurable, or for which measurement 
is impractical, may be classified as LOE. 


LOE activities are of no benefit to a manager using EVMS because they 
measure the passage of time versus the accomplishment of tasks or delivery of products. 
While some activities clearly fall in the realm of LOE, minimizing the categorization of 
activities as LOE is necessary for a manager to accurately measure and manage his 


program’s health. 


LM Aero provides little guidance in the area of controlling LOE. They 
merely state that it “will exist only for those tasks where discreet or apportioned work 
measurement techniques cannot be effectively applied” and “will be separated from 
discreet and apportioned effort at the work package level” (Lockheed Martin Aeronautics 


Corporation, 2005). 


This limited discussion of LOE and lack of specific guidance was apparent 
when assessing LM Aero’s approach to applying LOE as a work measurement approach 
for Spiral 2 contracts. While it was expected that some efforts such as scheduling, 
configuration management and program management activities would naturally be 
associated with LOE, the use of LOE in Spiral 2 contracts did not appear to be in line 
with this criterion’s goal of minimizing its use. One example was the use of LOE for 
some rework activities associated with the development of requirements and design 
documentation during the first two Spiral 2 contracts. Because of the difficulty of 


defining the activity associated with the rework of documentation during review cycles, 
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LOE was used to capture this effort. This is just one example where LOE could have 
been better defined as discreet work, indicating that LM Aero could better focus their 
EVMS on minimizing LOE. Examples such as this led to a marginal rating for this 


criterion associated with control of LOE. 


h. EVM Criterion #13: Establish overhead budgets for each 
significant organizational component of the company for 
expenses that will become indirect costs. Reflect in the program 
budgets, at the appropriate level, the amounts in overhead pools 
that are planned to be allocated to the program as indirect costs. 


This criterion highlights concerns regarding the proper allocation of 
indirect costs to a project or program. To preclude manipulation a company must specify 
areas of indirect cost at program inception, with formal internal controls directing any 
subsequent changes. When indirect costs do not allocate directly to control accounts, the 
contracted organization should indicate some point within the WBS where the indirect 


costs apply. 


LM Aero policy stipulated a formal, annual establishment of overhead 
budgets plant-wide. To start with, LM Aero forecasted both known and estimated 
business for their next fiscal cycle. This business included integration of the annual 
overhead budget with plans for contract performance, sales and profits, capital 
investments, and cash flow requirements. The Overhead Section of the Aeronautics 
Controller would then internally publish the indirect manpower and dollar targets 
necessary to support the assumptions used.2! Following that, overhead budgets were 
established with functional organization development of internal assessments of 
requirements Targets were developed based on historical trends, current spending levels, 
expected or known changes in task/requirements and other quantitative or qualitative data 


and assigned based on the Aeronautics Controller Overhead Section’s assessment. 


Discrete items of cost were assigned to identified organizations for 


planning and control (starting with indirect manpower) and formally and discretely 


21 The assumptions used for overhead planning were not made available. 
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identified to the appropriate burden center/ overhead pool (see Figure 6 for a list of 
typically allocated overhead expenses; see Figure 7 for a generic example of how discrete 
expense accounts are allocated to overhead pools at LM Aero). Once approved, the 
budgeting system allowed for adjustments to the overhead budget due to anticipated 
changes in conditions and/or assumptions. Any overhead budget adjustments, however, 
required formal requests “from proper line management” (Lockheed Martin Aeronautics 
Corporation, 2005). Properly requested adjustments required detailed justification, which 
were evaluated by the Aeronautics Controller. LM Aero concluded with the statement 
that “appropriate line and company managers” must approve all upward adjustments 


(Lockheed Martin Aeronautics Corporation, 2005). 


LM-Aero Overhead 
Accounts 


Salaries & Wages 

Employee Awards 

Group Insurance Program 
Payroll Taxes & Insurance 
Fringe Benefits 

Retirement Plan 

Savings Plan 

Misc. Employee Benefits 
Indirect Supplies & Material 
Utilities 

Depreciation & Amortization 
Taxes 

Travel & Communication 
Expense 

Miscellaneous Expenses 
Proposal & Bidding Expense 
Independent Research & 
Development 

Intercompany Expense 





Figure 6. | Summary of Company Overhead Expense Accounts 


(From: Lockheed Martin Aeronautics Company EVMS Description, 2005) 
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Figure 7. Overhead Pool Assignment for Organizational Expense 


(From: Lockheed Martin Aeronautics Company EVMS Description, 2005) 


In practice, the ability to manipulate indirect costs between profitable and 
less profitable programs creates a heightened concern to government procurement 
offices. The Company’s annual publishing of its EVMS Description and Cost 
Accounting Standards created the expectation that LM Aero adhered to industry- 
acceptable methods of indirect cost allocation. Moreover, tracking and auditing for the 
allocation and accumulation of indirect costs against F-22 contracts was managed by 
DCMA located in the LM Aero facilities in Fort Worth, Texas. DCMA is responsible for 
negotiating and monitoring overhead rates for all LM Aero contract efforts based in Fort 
Worth, including Spiral 2 software development. Additionally, they received and 
monitored all EVMS reports for the Spiral 2 effort, and identified no negative findings or 
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inconsistencies with how LM Aero allocated, accumulated or reported indirect costs on 
the Spiral 2 contracts. As such, with respect to Spiral 2’s allocation of indirect costs 
towards authorized budget, the EVMS program warranted a satisfactory rating for 


meeting the purpose of this criterion. 


i. EVM Criterion #14: Identify management reserves and 
undistributed budget. 


Identifying and controlling both management reserve (MR) and 
undistributed budget (UB) is necessary to maintain the integrity of a program’s EVMS. 
MR, used to cover the cost of “unknowns”, must be held outside the PMB and will not be 
assigned to a WBS element until a decision is made by management to do so. UB, part of 
the project PMB, represents funds that have been identified as essential for completion of 


the project, but have yet to be assigned to a WBS element. 


While LM Aero policy leaves it to the discretion of the program manager 
whether MR is required for a project, it is clear in directing that MR and UB be 
maintained separate from the PMB. Further, it clearly stipulates that all transactions with 


respect to MR and UB will be documented. 


All three Spiral 2 contracts had both MR and UB clearly identified at the 
IBRs. With all three efforts being initiated as UCAs there was UB for all three contracts. 
Once the efforts were negotiated, the UB was appropriately distributed to the applicable 
WBS elements. The assessment for this criterion associated with MR and UB was 


determined to be excellent. 


J EVM Criterion #15: Provide that the program target cost goal is 
reconciled with the sum of all internal program budgets and 
management reserves. 


This criterion concentrates on the accountability of all project funds. Asa 
result, a contractor must strive to keep its total project costs within their authorized 
budgets. Along those same lines, the contractor must exhibit documented control 
processes to ensure that total project costs are kept in check. 


ST 


LM Aero’s response to this criterion meets expectations. The negotiated 
contract target cost was distributed to the control accounts—$36M total. The total target 
cost ($46M) also included UB ($5M) and MR ($5M). Therefore, the amount distributed 
to the control accounts, plus the value of UB and MR, reconciled to the contract's total 


target cost. 


Today’s cost tracking software makes it difficult—if not impossible—to 
fail in observing this criterion. When the F-22 SPO received CPRs from LM Aero 
electronically, the CPRs were accompanied with software packets that ported directly 
into a program made expressly for tracking EV data. Therefore, should any portion of 
program costs not equate to the total, the software identified and isolated the cost(s) in 
question. This made reconciliation more of a technical issue rather than an analytical 
one—in most cases. In conclusion, with respect to Spiral 2’s reconciliation of all internal 
program budgets (plus reserves) the EVMS program warranted an excellent rating for 


meeting and exceeding the purpose of this criterion. 


J Group 3: Accounting Criteria 


Table 4 identifies the assessments provided to each of the six ANSI/EIA EVMS 


Accounting Criteria. 


Criteria Assessment 


Record direct costs Excellent 
Summarize direct cost to the WBS 
Summarize direct cost to the organization 





Record indirect costs Excellent 
Identify unit/lot costs Inconclusive 
Record material costs Inconclusive 
Table 4. | Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Accounting 
Criteria 
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a. EVM Criterion #16: Record direct costs in a manner consistent 
with the budgets in a formal system controlled by the general 
books of account. 


According to Fleming and Koppelman (2000, p171), the preferred method 
for recording direct costs is “applied direct method”, accounting for resources as they are 
used or consumed. In labor intensive efforts such as software development, where teams 
are typically established by functional disciplines, the challenge is to ensure direct labor 


costs are appropriately charged to the correct project with many. 


LM Aero policy provides great detail regarding the cost identification and 
numbering systems, work order nomenclature, direct charge policies, work-in-process 
(WIP) subaccounts, recurring vs. nonrecurring costs, and the direct labor 
charge/accounting process. LM Aero uses the applied direct method, based on the 


procedures outlined in LM Aero policy. 


Although not unique to F-22 Spiral 2 software development (or even to 
software development in general), the government has always had concerns in the area of 
cost accounting and how charges are tracked to different work packages. With managers 
or IPTs having more than one charge account at their disposal, on any given day, what 
prevents an individual or IPT from charging to a “healthy” account versus the one they 
are working on that is near or at an over-run state? Although this is not addressed in LM 
Aero policy, the government must continue to rely on the power of audits, DCMA, and 
LM Aero adherence to generally accepted accounting standards. The detail provided by 
LM Aero procedures along with observed implementation warranted an excellent rating 


for this criterion associated with recording direct costs. 


b. EVM Criterion #17: When a WBS is used, summarize direct 
costs from control accounts in the WBS without allocation of a 
single control account to two or more WBS elements. 


The purpose of this criterion: to eliminate the confusion resulting from 
multiple WBS elements crisscrossed with multiple control accounts. A WBS element by 


its nature identifies a unique control account, which eliminates the possibility of dividing 
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a control account between multiple WBS elements. Additionally, the WBS design 
ensures that a lower level element uniquely identifies with one (and only one) higher 


level element or tier. 


The LM Aero F-22 team strived to meet this criterion via its basic 
accounting numbering system. An eight character work order forms the foundation of the 
system. Spiral 2 work orders, through their unique eight characters, recorded and 
identified incurred costs to the contract, then to the contract line item, followed by work 
breakdown structure elements, and finally to the discrete tasks within the WBS elements. 
Work order numbers were established by the Accounting Department within the terms of 
the Spiral 2 contract modifications to the REDI contract. A record of all authorized work 
orders was maintained by the Accounting Department, while active work orders were 


maintained in a computer file which was readily accessible by company personnel. 


In practice, LM Aero identified the first three characters of the work order 
as representing the contract code number assigned to the contract. With respect to Spiral 
2, this code was shared with all REDI contract actions and subsequent modifications. 
The second grouping of the work order (characters four and five), also known as the 
project code, aligned costs with CWBS elements. The third and final grouping of the 
work order (characters six through eight), also known as the job code, provided detail 
within the project by identifying costs along with their respective job or specific task 
item. With respect to Spiral 2’s WBS utilization and respective control accounts related 
to one and only one given WBS, the EVMS program warranted a satisfactory rating for 


meeting the purpose of this criterion. 


Cc. EVM Criterion #18: Summarize direct costs from the control 
accounts in the contractor’s organizational elements without 
allocation of a_ single control account to two or more 
organizational elements. 


This criterion builds on Criterion #2 which identified the need for 
assigning each element of the WBS to an organization. Criterion #18 requires a 


contractor’s cost accounting system be capable of collecting, summing and reporting cost 
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accounts by functional organizations (e.g., engineering, quality, production, etc.). This 
provides managers an ability to measure the work being accomplished by functional area 
as a program progresses. This criterion also explicitly stipulates what is inferred in 
Criterion #2— that a single control account may not be assigned more than one functional 


organization. 


As with most of today’s contractors, LM Aero does most of their 
development activities through multi-functional IPTs. Although this criterion appears to 
call for breaking cost/schedule account work packages apart based on functional 
organizations, LM Aero policy does not take this approach. In order to ensure the ability 
to summarize costs by functional organizations, LM Aero developed an employee 
numbering approach that identifies each employee by their functional area regardless of 
what organization to which they are assigned. Since employees record their hours in a 
work package using their employee number, a summary of labor costs can be 


accomplished by functional area. 


Spiral 2 used many functional disciplines during its development. The 
ability to identify/summarize effort by functional organization was never observed by the 
government. Although LM Aero policy identifies this as a capability, reports showing 
this capability were never produced. Government requests for these reports were never 
supported leaving questions regarding LM Aero’s ability to meet this criterion. Based on 
the data provided, however, there was enough concern to warrant a marginal rating for 


this criterion. 


d. EVM Criterion #19: Record all indirect costs that will be 
allocated to the contract. 


EVM Criterion #19 is related to the functional responsibility for 
controlling indirect costs of Criterion #4. This criterion, however, goes further in 
requiring that the contractor be able to identify indirect costs at the point charged, 
summarize them, and relate them to original planned budgets. It also requires the 


relationship be formally documented between those controlling indirect costs and those 
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able to incur costs against indirect budgets. Whatever method is chosen by the contractor 


to allocate indirect costs, it must be documented, consistently applied and auditable. 


LM Aero policy regarding the allocation and accumulation is consistent 
with generally accepted accounting procedures and uses a monthly adjusted year-to-date 
approach that is intended to minimize year-end adjustments. Additionally, the policy 
identifies the requirement to accumulate indirect costs both by expense account and 


organizational department, consistent with this and other criteria. 


Tracking and auditing for the allocation and accumulation of indirect costs 
against F-22 contracts is managed by DCMA located in the LM Aero facilities in Fort 
Worth, Texas. DCMA is responsible for negotiating and monitoring overhead rates for 
all LM Aero contract efforts based in Fort Worth, including Spiral 2 software 
development. Additionally, they received and monitored all EVMS reports for the Spiral 
2 effort, and identified no negative findings or inconsistencies with how LM Aero 


allocated, accumulated or reported indirect costs on the Spiral 2 contracts. 


e. EVM Criterion #20: Identify unit costs, equivalent unit costs, or 
lot costs when needed. 


The focus of this criterion is the establishment of unit, lot, and recurring 
costs for use in future efforts. In order to accomplish this, the contractor must be able to 
distinguish, in cost accounts, the differences between recurring (e.g., production) and 


non-recurring (e.g., development) activities. 


LM Aero cost accounting does not support the tracking of direct unit or lot 
costs. The method used to obtain these values is an annual calculation of average unit 
cost for the specified WBS elements. Segregation of recurring and non-recurring, LM 
Aero establishes discreet work accounts that are defined based on recurring or non- 
recurring activities. The charges associated with these efforts can be summarized 


through the WBS as with other work activities. 


Spiral 2 software development for the F-22 was almost exclusively a non- 


recurring effort. All “production” of Spiral 2 software was non-recurring and the 
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installation of the finished software to aircraft was accomplished via separate production 
contracts (or modifications). While there were opportunities to apply lessons learned to 
follow-on F-22 development activities, these could not be considered recurring as they 
involved different requirements, schedules and teams. Based on the lack of applicability 


to Spiral 2, this criterion was assessed as inconclusive. 


f EVM Criterion #21: For EVMS, the material accounting system 
will provide for: 


1. Accurate cost accumulation and assignment of costs to 
control accounts in a manner consistent with the budgets using 
recognized, acceptable, costing techniques. 


2s Cost performance measurement at the point in time most 
suitable for the category of material involved, but no earlier than 
the time of progress payments or actual receipt of material. 


5. Full accountability of all material purchased for the 
program including the residual inventory. 


This criterion ensures useful measurements of cost and schedule variance 
(CV and SV) related to the material accounting system. It requires allocation of all 
appropriate purchases to the same accounting period, thus reflecting planned versus 
earned value (proper recording enables an accurate SV). This same expected allocation 
process also aids in the proper accounting of earned versus actual costs (proper recording 


enables an accurate CV). 


LM Aero provided a detailed (but general) description of the company’s 
in-place processes that deal directly with this criterion. LM Aero’s Accounting division 
established routines that ensured the validity of the data input (used for tracking cost 
accumulation) while enabling any necessary editing for “transactional existence and 
compatibility” (Lockheed Martin Aeronautics Company, 2005). These same routines 
also helped maintain appropriate records: specifically, those records dealing with 
requirements, commitments, receipts, issues, and inventory by group, part number, unit, 
and actual price. The accounting records also enabled identification of different groups 
or cost types, which in turn permitted summarization of costs into basic categories such 


as raw materials, hardware, equipment, tooling materials, and purchased parts. Finally, 
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the accounting records also allowed for subcontract and inter-company work transfers 


identification. 


The description then began an overview of direct charge materials 


(procurements), work-in-process accounting, and (contract) inventories: 


Upon receipt, raw materials, hardware, equipment, tooling material, and 
purchased parts, along with major component/subsystem procurements, 
were charged to the Spiral 2-specific contract work order. Materials- 
related items such as tooling and shipping (1.e., other than manufacturing 
materials) were reported against the gaining control accounts at issue from 


inventory. 


Sub-accounts identified as work-in-process (WIP) provided the status of a 
given item in the process flow (e.g., on dock, in inventory, or placed into 
production). WIP-related progress payments were segregated into 
separate WIP-inventory accounts—under the buying contract—for unique 
cost identification. Upon receipt of the procured item, the subcontractor's 
progress payments were liquidated and the value was recorded to the 


appropriate WIP account. 


Contract inventories were carried at purchase order price. Source 
documents from these contract inventories were utilized to collect charges 
for input to the cost accumulation system (from receiving reports, 
invoices, requisitions, etc). From that point, costs were accrued for 
unbilled received items and unmatched invoice suspense items, such as 
those related to subsystem vendors and subcontractors. These accruals 
were distributed to the appropriate contract work orders and work-in- 


process sub-accounts each month. 


In practice, incurred or accrued costs for direct charge materials and major 


components/subsystems procurement were reported against the benefiting control 


accounts upon issue from inventory for performance measurement purposes. The study 


failed to acquire information regarding LM Aero’s material accounting system as it 


pertained directly to Spiral 2. LM Aero policy meets the guidelines of this criterion. The 
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apparent lack of visibility, however, into the Spiral 2-specific material accounting system 
coupled with LM Aero’s vague, unbilled items policies and unmatched invoices made an 
unchallenged acceptance of criterion satisfaction with respect to Spiral 2 difficult. With 
respect to Spiral 2 existence within LM Aero’s material accounting system, the EVMS 


program warranted an inconclusive rating due to lack of Spiral 2-specific supporting data. 


4. Group 4: Analysis Criteria 


Table 5 lists the assessments determined for each of the six ANSI/EIA EVMS 
Analysis Criteria. 


Criteria Assessment 


Identify schedule and cost variances 
Analyze schedule and cost variances 


Analyze indirect costs 
Summarize data elements and variances for reporting 
Implement managerial actions 
Develop revised estimates of cost at completion 
Table 5. | Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Analysis 
Criteria 





a. EVM Criterion #22: At least on a monthly basis, generate the 
following information at the control account and other levels, as 
necessary for management control, using actual cost data from, 
or reconcilable with, the accounting system: 


1. Comparison of the amount of planned budget and the 
amount of budget earned for work accomplished. This 
comparison provides the SV. 


2 Comparison of the amount of budget earned and the 
actual (applied where appropriate) direct costs for the same work. 
This comparison provides the CV. 


This criterion forms the foundation of EVM reporting. Its focus is to 
compare performance at the control account level with earned value results. Compliance 
with this criterion should translate into program managers identifying potential overruns 


and underruns. 
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LM Aero stated that BCWS, BCWP and ACWP were identified for each 
control account monthly. The Accounting Department’s Cost Ledger provided the 
ACWP for each control account. BCWS values were generated from work packages 
according to the PMB, and summarized to the control accounts for each respective cost 
element. Budgetary values for cost elements reported as earned (for completed work 
packages) and completed portions of in-process work packages resulted in the BCWP. 
For work packages that utilized other-than-cost work measurement systems, target values 
(e.g., standard hours) assigned to activities were earned as activities completed. 
Comparing the cumulative earned targets against the total target value for each 
performing department, the work package percent completion status is determined and 
used to calculate BCWP for the work package.22, BCWP for special cases was calculated 
as follows: 

e Work packages established for tooling and manufacturing materials and 
work packages established for procured tools earned their budgetary dollar 
value incrementally as these materials or tools were issued for processing 


and/or use. 


° The cost of any subcontracted items/systems issued was recorded to 
specific accounting WIP sub-accounts by work order. Cumulative actual 
costs against these sub-accounts were compared to the total estimate for 
these subcontracted items/systems to determine a percent completion then 
used in calculating a subcontractor BCWP. Budgets for the nonrecurring 
effort of major subcontractors were time-phased within work packages 
according to the planned receipt and payment of each vendor's invoice: 
when the invoice was allocated to WIP, earned value (BCWP) was 


awarded. 


e Major subcontractors classified as critical subcontractors (e.g., Boeing, 


Northrop-Grumman) and under contract for other than a firm fixed price 


22 For many discretely measured work packages, the timing and amount of the 
budgetary value earned depends upon activity completion. As an activity completes, 
BCWP was eared for the work package in the proportion of the activity's resource 
estimate (relative to the total resource estimate of the work package). 
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were contractually obligated to comply with additional control and 
reporting criteria. For example, status reporting in compliance with ANSI 
EVMS Criteria provided an additional cost and schedule performance 
measurement tool. This data became the source of any reported 


performance related to the subcontractor. 


In practice, as stated within the description of this criterion, performance 
data was essential at the control account level since it effectively enabled monitoring of 
project performance. For a project like Spiral 2 that was organized according to an IPT 
structure, the performance data provided the program manager with a summary of 
progress and cost performance on each WBS element assigned to the Spiral 2 team. 
Spiral 2 data generated at month-end began reflecting a favorable performance variance, 
confirming the underrun anticipated due to the over-estimated undefinitized contract 
much of Spiral 2 existed under. During the Spiral 2 Phase B CPR reporting period (April 
2004 through January 2005), the F-22 SPO had no outstanding issues regarding the 
figures generated for BCWS, BCWP, ACWP, CV, and SV, and LM Aero consistently 
and reliably reported them every month during the contract duration.23 In conclusion, 
with respect to Spiral 2’s generation of BCWS, BCWP, ACWP, CV, and SV on at least a 
monthly basis, the EVMS program warranted a satisfactory rating for meeting the 


purpose of this criterion. 


b. EVM Criterion #23: Identify, at least monthly, the significant 
differences between both planned and actual schedule 
performance and planned and actual cost performance, and 
provide the reasons for the variances in the detail needed by 
program management. 


This criterion asserts that whenever either SV or CV reported from a given 
CPR exceeds a previously agreed-upon level between customer and contractor, the 


contractor should analyze associated drivers and provide a reason why that threshold was 


23 This criterion does not consider the disconnect between undefinitized and 
definitized contract costs: rather, it only looks for the successful generation of the EVM 
measures listed. 
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broken. This arrangement should also filter down to major-subcontractors. Additionally, 


a plan for recovery is considered customary alongside a given variance analysis. 


LM Aero stated that contract significant variances were determined by the 
variance reporting conditions negotiated for the CPR. Whenever a WBS summary level 
element's variance satisfied the conditions negotiated for CPR analysis (i.e., negatively 
surpassed the contracted threshold), the company documented an analysis of variances 
for those control accounts principally responsible for the summary level variance, and 
inserted a variance package within that period’s CPR (if the customer contracted for it). 


Reasons for significant progress differences from the plan were also identified. 


The CPRs related to Spiral 2 Phase B reflected adherence to this criterion. 
When Spiral 2 Phase B was contracted, the F-22 Program Office identified the CPR 
Format 5 (Variance Analysis) as necessary for effective program oversight. Every month 
LM Aero was contractually required to report variance analysis. Beginning with the 
October 2004 CPR, LM Aero reported significantly positive CVs, indicating the already- 
anticipated underrun derived from definitization of the project at a much lower target 
cost. In conclusion, with respect to Spiral 2’s identification of significant CV and SV on 
at least a monthly basis, the EVMS program warranted a satisfactory rating for meeting 


the purpose of this criterion. 


Cc. EVM Criterion #24: Identify budgeted and applied (or actual) 
indirect costs at the level and frequency needed by management 
for effective control, along with the reasons for any significant 
variances. 


Changes in indirect costs can be an important consideration of the project 
management. An increase in indirect charges is driven by either an increase in the 
indirect expenses of the project or a decrease in the direct base over which the indirect 
charges are applied. This criterion requires that changes against the baseline for indirect 


charges be identified and adverse impacts be addressed. 


The identification of variances between indirect budget and actual indirect 


charges is evaluated on a monthly basis by each LM Aero department head. 
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Additionally, indirect manpower actuals are collected on a weekly basis and variances are 


reported to upper management. 


Although the F-22 Program Office was never able to obtain detailed 
indirect charges associated with modernization contracts, DCMA who is responsible for 
monitoring all indirect charges per negotiated rates never identified any finding or 
concerns for the Spiral 2 delivery orders. Based on this absence of negative findings an 


assessment of satisfactory was provided for meeting the purpose of this criterion. 


d. EVM Criterion #25: Summarize the data elements and 
associated variances through the program organization and/or 
WBS to support management needs and any customer reporting 
specified in the contract. 


This criterion acknowledges control account-level variances are not 
reported simply because they either offset themselves (a negative and a positive) and/or 
the contractor can (and prefers to) handle such management details in-house. However, 
any project must have flexibility in reporting variances. Furthermore, internal and 


external reports must align. 


LM Aero stated that performance data was summarized from the control 
accounts through the WBS. Also, data from one control account was allocated only to its 
unique summary-level WBS element. BCWS, BCWP, ACWP, SV, and CV were 


summarized directly to the reporting level specified within the contract in question. 


Once again, the CPRs related to Spiral 2 Phase B reflected adherence to 
this criterion. When Spiral 2 Phase B was contracted, the SPO-side of the F-22 program 
identified the CPR Formats 1 (WBS), 2 (IPT Structure), and 5 (Variance Analysis) as 
necessary for effective program oversight through the Spiral 2 Phase B-associated CDRL. 
Thus, every month LM Aero was contractually required to report data elements and 
variance analysis within the parameters of these formats. In conclusion, with respect to 
Spiral 2 summarization of data elements (and associated variances) through the program 
organization and/or WBS that supported management and customer needs, the EVMS 


program warranted a satisfactory rating for meeting the purpose of this criterion. 
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e. EVM Criterion #26: Implement managerial actions taken as the 
result of EV information. 


The intent of this criterion is to ensure specific procedures and policies are 
set in place to ensure management identifies corrective actions whenever EV variances 
indicate either poor performance (i.e., negative variances) or a faulty baseline plan (i.e., 
positive variances). Thresholds must be identified in advance to trigger management 
involvement. These thresholds must be meaningful, should be at multiple monitoring 


points, and be in terms of both positive and negative variances. 


LM Aero considers the requirement of this criterion to be synonymous 
with Criterion #22, generation of monthly reports identifying schedule and cost 
variances. The identification of predefined variance thresholds are negotiated for each 
contract LM Aero enters with the government. According to LM Aero procedures, once 
these thresholds are breached, actions must be taken to both identify the reason for 


significant variances and identify the managerial corrective action. 


When looking at Spiral 2 DO 0002, the only stand-alone Spiral 2 contract 
complete to date, compliance with this criterion was suspect. Although variance 
thresholds were identified (monthly: $1M and +/- 10%; cumulative: $2M and +/- 10%) 
there was no evidence that these thresholds meant anything. Throughout the entire effort 
DO 0002 showed significant positive cost variances. Five of the ten months of EV 
reporting had cost variances greater than the $1M threshold set by the contract. This 
indicated a questionable contract baseline. Although various root causes were discussed 
for the significant variances, no steps were taken to rebaseline the program to a more 
realistic plan. At its completion, DO-0002 completed approximately 40 percent under its 
UCA value, clearly indicating the PMB managed to during the execution of the contract 
was not an accurate reflection of contract costs. In conclusion, the assessment for the F- 
22 EVMS implementation of the criterion associated with taking necessary managerial 


actions was insufficient. 
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f EVM Criterion #27 — Develop revised estimates of cost at 
completion based on performance to date, commitment values for 
material, and estimates for future conditions. Compare this 
information with the PMB to identify variances at completion 
important to company management and any applicable 
customer-reporting requirements, including statements of 
funding requirements. 


This criterion deals with the EVM summary of data elements known as 
EAC.24 Routine calculation of EAC must ensure both accuracy and timeliness, as 
transgression of either brings the entire cost of the project into uncertainty. The final step 
is to routinely compare the EAC with the PMB to ascertain the progress, or lack thereof, 


of the program in question. 


LM Aero policy required performing comprehensive updates of cost-to- 
complete at least twice a year, and more frequently if directed. The initial step in 
preparing an estimate of cost-at-completion was ensuring all authorized tasks were 
aligned with both their appropriate WBS element(s) and their respective departments 
(those expected to perform the tasks). Estimated completion dates were reviewed and 
revised as appropriate, with consideration given to performance to date, and authorized 
tasks not yet defined and planned as specific activities were forecast within undefined 
aggregates. Notable within cost-to-complete forecasts (due to management and customer 
interest) were direct labor and overhead rates. The direct labor rates included projections 
made from extrapolations of rates-to-date, labor union agreements, company merit 
assumptions, changes projected in level of employment, and skill mixes required to 
complete the remaining work. Cost-to-complete overhead rates were expressed as 
applied overhead rates and forward pricing rates. They were normally revised annually 
(or more frequently) based on actual and projected business conditions. Once initial 
calculation of cost-to-complete was finished, the project team determined the project's 


cost-to-complete iteratively via process of estimation, review, feedback, and revision. 


24 See discussion within the EVM Primer section of Chapter I. 
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The CPRs related to Spiral 2 Phase B reflected adherence to the 
development of revised estimates portion of this criterion. CPR Formats 1 and 2 
highlighted Latest Revised Estimates (LREs) as the project went on, showing a slight 
positive CV until the final report (January 2005) formally acknowledged the definitized 
contract price via the Management EAC block in Format 1. It is assumed this late 
acknowledgement of such a significant variance was forced due to attempted adherence 
to a somewhat-conflicting Criterion #30. The revised estimates provided little to no 
management control given the magnitude of the contract price change, and thus at least 
partly failed to meet the comparison portion of this criterion. The fact that useful, formal 
information was not reported until the final CPR validated that statement. In conclusion, 
with respect to Spiral 2 development of revised estimates of cost at completion based on 
performance to date, commitment values for material, and estimates for future conditions, 
the EVMS program warranted a marginal rating for exhibiting some material failures in 


meeting the purpose of this criterion. 


5. Revisions Criteria 


Table 6 identifies the assessment for each of the five criteria under the ANSI/EIA 
EVMS Revisions Criteria. 


Criteria Assessment 


Incorporate changes into plans, budgets and schedules 
Reconcile budgets changes Inconclusive 


Control retroactive changes Excellent 
Control revisions to the program budget Excellent 
Document changes to the PMB 
Table 6. | Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Revisions 
Criteria 
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a. EVM Criterion #28: Incorporate authorized changes in a timely 
manner, recording the effects of such changes in budgets and 
schedules. In the directed effort prior to negotiation of change, 
base such revisions on the amount estimated and budgeted to the 
program organizations. 


Although “timely” is not defined, it is reasonable to expect any contractor 
and government team should be able to agree on what constitutes a “timely” update to 
project plans to incorporate necessary changes. Changes may be driven either internally 
(e.g., significant cost or schedule overruns) or externally (e.g., changes in contract scope 
or available funding). Regardless of the cause, the owners of the PMB must be able to 
update the plan and incorporate the necessary changes in a short enough time to minimize 


lack of useful EV data. 


The LM Aero policy governing this criterion stipulates the contract budget 
base will be updated following contractual authorization. Although no specific timeline 
can be provided for the various contract modifications that could occur, the policy states 
that “generally within 60 days after contractual authorization the change will be 
incorporated into program schedules and the performance measurement baseline” 
(Lockheed Martin Aeronautics Corporation, 2005). Although the policy addresses 
changes driven by contract (external) changes, there is no discussion or direction 
regarding the need to rebaseline the PMB due to internal project execution issues. This is 


addressed in Criterion #32. 


Rebaselining of schedules and PMB has been a challenge for the F-22 
since the earliest days of the program. Based on F-22 records, the entire development 
program underwent annual rebaselining the first three years of the program. This set the 
tone for the remainder of the EMD phase of the program and continued into the post- 
EMD modernization development efforts such as Spiral 2. According to interviews with 
Spiral 2 managers, the approach regarding Spiral 2 was to use “rolling baselines” to 
accommodate constantly changing schedules, requirements and external perturbations. 
While these frequent changes to the baseline may be occurring in a “timely” manner, the 
intent of this criterion is not to merely update the PMB. It is to accurately and 


realistically update the PMB so that frequent updates of the program plan are not 
73 


required. The challenge for the Spiral 2 management team was discerning when PMB 
rebaselines were being driven by external factors versus poor performance against the 
baseline. The concerns associated with this criterion were not driven by the Spiral 2 
implementation of the criterion as much as with the factors that contributed to the 


frequent rebaselining; therefore, a rating of satisfactory was warranted for this criterion. 


b. EVM Criterion #29: Reconcile current budgets to prior budgets, 
in terms of changes to the authorized work and internal 
replanning in the detail needed by management for effective 
control. 


This criterion highlights the importance of traceability with respect to 
WBS, specifically changes affecting the baseline. Program teams need to provide this 
traceability to the lowest level, given that baselines are generally developed with a 
bottoms-up method. That traceability provides the means for reconciliation between 


current and prior budgets. 


LM Aero asserts that any of their programs’ contract budget bases and/or 
PMBs will change only as a result of negotiations that result in contractual change 
authorizations or revisions to proposal values. (PMBs can also change as a result of 
approved internal replanning). LM Aero identifies each contractual change and 
reconciles it to the original contract budget base and/or PMB. This ensures target cost 


integrity reporting both internally and to its customers. 


The short duration of Spiral 2 DO 0002 appeared to negate any 
opportunity that might have provided proof of adherence to this criterion. By January 
2005, when the change to the total contract budget was formally recognized, $22M of the 
revised contract total of $24M had already been accomplished. Thus, per the Spiral 2 DO 
0002 CDRL, with over 95 percent of the work accomplished, CPRs were no longer 
required.25 The study failed to acquire information regarding LM Aero’s traceability of 


budget changes as it pertained directly to Spiral 2. Though LM Aero policy met the 


i 25 $22M was 95 percent of the non-award fee total of the definitized contract budget 
ase. 
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guidelines of this criterion, the apparent lack of visibility into Spiral 2-specific budgetary 
changes made an unchallenged acceptance of criterion satisfaction with respect to Spiral 
2 alone difficult. As such, with respect to Spiral 2’s ability to reconcile budgetary 
changes, the EVMS program warranted an inconclusive rating due to lack of Spiral 2- 


specific supporting data. 


EVM Criterion #30: Control retroactive changes to records 
pertaining to work performed that would change previously 
reported amounts for actual costs, EV, or budgets. Adjustments 
should be made only for correction of errors, routine accounting 
adjustments, effects of customer- or management-directed 
changes, or to improve the baseline integrity and accuracy of 
performance measurement data. 


The intent of this criterion is to ensure the integrity of the EVMS. If 
budgets or actuals are changed after-the-fact, the usefulness of the EVMS as a 
management control tool is lost. The only appropriate retroactive updates are those made 


to correct errors or other legitimate accounting adjustments. 


It was not surprising to find LM Aero policy specifically prohibits changes 
to previously reported actual costs unless it is done as a correction of errors or an 
accounting adjustment. Even then, the policy stipulates any changes to previously 


reported EV data must be coordinated through and approved by senior management. 


Inappropriate changes to reported data would be a serious infraction of 
contractual obligations and a breach of trust. There has never been an indication through 
audits or otherwise that LM Aero would jeopardize their relationship with the 
government or their EVMS certification to retroactively change EV data. As such an 


excellent rating was warranted for this criterion. 


d. EVM Criterion #31 — Prevent revisions to the program budget, 
except for authorized changes. 


Virtually all DoD programs experience challenges to the plan originally 


put forward at the beginning of the effort. Sometimes these challenges become so severe 


do 


that project managers need to change the budget baseline to a more realistic budget 
baseline. This criterion recognizes baselines may change, but any changes to the budget 
associated with the baseline must be changed on a limited basis and in a controlled 


environment. 


In the case of LM Aero, EVMS policy states that a contract budget base 
can only be changed when authorized through a proposal update. This policy ensures the 
contractor will never unilaterally change the budget baseline. Based on this strong policy 
and observed performance in this area, an excellent rating was warranted for this 


criterion. 


e. EVM Criterion #32: Document changes to the PMB. 


As changes occur to the project PMB, these changes must be controlled 
and traceable. Unauthorized changes to the PMB undermine the utility of the EVMS. 
Similarly, when an update to the baseline cannot be traced to the original plan, it becomes 
difficult to identify where trade-offs occurred, impacts to other areas of the program, or 


simply program history. 


Under Criterion #10, LM Aero typical policy is to define detailed work 
packages for all efforts within the current contract year. Budgets for activities planned 
beyond the current year, particularly those in support of development efforts, are held in 
planning packages until more is known regarding the activity’s details. As the project 
progresses and these planning packages are transitioned into detailed work packages, 
these changes are captured in documentation, showing the relationships between the 
planning packages and work packages. Additionally, changes driven by external 
program pressures, as discussed in Criterion #28, are to be documented and identify 


linkages to the previous PMB. 


As discussed in Criterion #28, the Spiral 2 PMB was updated using 
“rolling baselines” to accommodate for external perturbations. Unfortunately, while 
these changing baselines showed linkages to previous PMB, where possible, many times 


the changes to the PMB were significant enough that traceability between activities 
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became very difficult. Additionally, it was difficult to identify which changes occurred 
in the PMB as a result of external pressures versus poor internal performance. The 
inability of project managers to identify these differences limits the utility of the EV data. 
These concerns notwithstanding, the F-22 documentation of changes to the PMB was 


assessed to be sufficiently in keeping with the criterion, warranting a satisfactory rating. 


C. SUMMARY 


Table 7 summarizes the authors’ assessments of how each of the 32 ANSI/EVMS 
criteria was applied on F-22 Spiral 2. While there were several criteria not met (or which 
lacked the requisite verification), there was not a systemic failure in the EVMS process at 
F-22. Rather, most of the challenges being faced by the use of EVMS on Spiral 2 were 
driven by issues outside of the EVMS spectrum. This will be further discussed in 
Chapter IV, F-22 EVMS Environment, where the F-22 development environment may 


have contributed to difficulties in EVMS implementation. 


fe: 


Criteria 


Group 1: Organization 


1 


Define authorized work (WBS elements) 
Identify organizational responsibilities 
Integrate the system 

Identify overhead management 

Provide for performance measurement 


Group 2: Planning, Scheduling and Budgeting 


Schedule the work 

Identify products, milestones and indicators 

Plan the Performance Measurement Baseline (PMB) 
Establish budgets for work 

Identify work packages 

Summarize work package budgets to control accounts 
Identify and control level of effort 

Establish overhead budgets 


Identify management reserves and undistributed budget. 


Summarize budgets to target cost 


Group 3: Accounting 


16 
17 
18 
19 
20 
21 


Record direct costs 

Summarize direct cost to the WBS 
Summarize direct cost to the organization 
Record indirect costs 

Identify unit/lot costs 

Record material costs 


Group 4: Analysis 


22 
23 
24 
25 
26 
27 


Identify schedule and cost variances 

Analyze schedule and cost variances 

Analyze indirect costs 

Summarize data elements and variances for reporting 
Implement managerial actions 

Develop revised estimates of cost at completion 


Group 5: Revisions 


28 
29 
30 
31 
32 


Table 7. 


Incorporate changes into plans, budgets and schedules 
Reconcile budgets changes 

Control retroactive changes 

Control revisions to the program budget 

Document changes to the PMB 


Assessment 


Excellent 


Inconclusive 


Excellent 
Excellent 


Excellent 


Excellent 
Inconclusive 
Inconclusive 


Inconclusive 
Excellent 
Excellent 





Assessment of F-22 Spiral 2 Implementation of ANSI/EIA EVMS Criteria 
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IV. F-22 EVMS ENVIRONMENT 


A. PREFACE 


While the previous chapter sought to answer if the F-22 EVMS was implemented 
per industry standards established for specifically that purpose, this chapter examines if 
the F-22 EVMS fulfills its role as a management control system for the avionics software 
development program in question and if F-22 managers used EVM products to manage 
the avionics development effort. In answering each of these questions, however, it was 
important to consider the environment within which the F-22 EVMS was employed. 
During the course of gathering data, collecting questionnaire responses, and discussing 
the use of EVM with F-22 managers, the authors identified certain foundational elements 
necessary for successful implementation of EVM not addressed by the 32 ANSI/EIA 
EVMS criteria. These are foundational issues behind any successful development 
program regardless of EVM use; however, many of these are often cited as the challenges 
facing today’s DoD software development efforts. Examples include requirements 
stability, schedule stability, funding stability and a realistic PMB. In the first section of 
this chapter three limiting factors or barriers were examined for their impact on the 
success of EVMS implementation in the F-22 Spiral 2 program. Next, the question “To 
what degree did the F-22 Spiral 2 implementation of EVM fulfill its role as a 
management control system for avionics software development?” was addressed by 
looking at CPRs for the avionics development program. Finally, if EVMS is to be 
valuable to managers, they must have confidence in the tool and be knowledgeable about 
EVM and the data being generated. The final section of this chapter reports the analysis 
of the questionnaire provided to Spiral 2 government and contractor personnel and 


intended to address this question. 


B. EVMS BARRIERS IN THE F-22 PROGRAM 
1. Undefinitized Contracts 


As discussed in Chapter II, Literature Review, at the end of EMD the F-22 
program recognized the challenges faced during EMD needed to be addressed in the 
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modernization program to prevent a repeat of cost overruns, schedule delays and 
undelivered performance. One of the approaches implemented to address the fluid 
requirements, constantly changing funding levels and technological challenges was the 
method of contracting for each incremental (“Spiral”) product delivery via phased 


contracts. 


Figure 8, Spiral 2 Summary Schedule, shows how each of these contracts was 
phased with respect to each other during the development of Spiral 2. The approach of 
having multiple contract phases for each spiral provided the desired programmatic 
flexibility; however, there was a cost to the business side of the program. With more 
contracts in work than ever before and the remaining issue of consistent changes in 


program baselines, the award of negotiated, definitized contracts became unattainable. 





Spiral 2 Available 
Contract to User on First 
©) Defirfitized Aircraft A 







9/30 








Figure 8. Spiral 2 Summary Schedule 
(From: DO 0019 Integrated Baseline Review) 


By the time Spiral 2 was initiated in 2003, the F-22 program frequently used 
UCAs in initiating new efforts with LM Aero. The complexities of the modernization 
program, with multiple increments at various stages all dealing with funding 
perturbations and requirement/technology iterations, made modernization even more 
susceptible to UCAs. Not an anomaly as much as the norm, Spiral 2 had all three of its 
development contracts initiated under UCAs. In fact, LM Aero completed the first two 
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phases of the Spiral 2 program before the respective contracts were definitized.”° 
Considering the undefinitized nature of UCAs,27 the use of EVM was extremely difficult, 


if not impossible. 


As discussed in the previous chapter, the second group of criteria deals with 
planning, scheduling and budgeting the program. If a program was initiated under a 
UCA, many times it was due to the immaturity of the program plan. Sometimes this was 
due to lack of understanding of the effort At times it was due to an inability to define this 
knowledge in terms of a negotiated definitive contract. Most times, as was the case with 


Spiral 2, it was a mixture of both. 


In some cases the scope of the work awarded under the UCA was defined so 
immaturely, it was impractical to hold an IBR early enough in the contract to capture a 
significant portion of the work to be performed. DO 0019 was an example of this with 
work being initiated on this effort in January 2005 under a limited UCA and the IBR 
being accomplished in January 2006. Clearly EVMS has limited value in instances such 
as these where an IBR can not be held to establish the baseline. When one year of effort 
(out of a total of three) was accomplished before an IBR was held, it severely limited the 


manager’s ability to measure progress against a program baseline. 


In the case of DO 0002, a program baseline was established via an IBR while 
under a UCA; however, there were significant limitations to the value of PMB 
established at this IBR. These limitations were driven by several factors, including: 


e A lack of confidence in the final contract value 


® Lack of buy-in between the parties on required tasks to complete the work 


(i.e., contract scope) 


26 Recall from Chapter II the first phase of Spiral 2, requirements analysis, was 
actually accomplished under REDI DO 0001, the overarching modernization program’s 
systems engineering and program management effort. This DO, initiated in January 2003 
via a UCA, was not definitized until after Spiral 2 scope was completed in March 2004. 
DO 0002, detailed design, was awarded via a UCA in March 2004 and was also 
definitized well after task completion in February 2005. DO 0019, awarded as a UCA in 
February 2005, was definitized in October 2005. 


27 Typically, UCA’s are based on not-to-exceed (NTE) values without much, if any, 
supporting data. 
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e The inability of the contractor to fully assign dollars to all Cost Account 
Managers (CAM). 


Regardless of whether an IBR was held and a program baseline established none 
of the PMB was likely to be contractual under a UCA. This resulted in difficulty for the 
government in controlling changes to the baseline during the execution of the UCA, or 


when the contract was later definitized. 


Spiral 2 DO 0002, detailed design, was an example of this challenge. Originally 
awarded as a UCA based on an NTE, DO 0002 IBR was accomplished less than 90 days 
after UCA award; however, with so much of the effort based on NTE values, it was 
difficult for government managers to assess the validity of the PMB during the IBR. 
Ultimately, DO 0002 would be completed under a UCA and was definitized after 
completion of work for 42 percent less dollars than originally contracted under the UCA. 
With this much of a difference between the work projected and the work performed, 
EVM was difficult to use as a management control tool. As the contractor team 
constantly ran below budget for the effort, government managers were always faced with 
the question of determining how much of the underrun was due to efficient performance 
versus excessive budget.28 This lack of confidence in the PMB significantly reduced its 


value as a management tool. 


2. Requirements Instability 


Requirements instability is one of the most commonly cited problems with 
suffering software development programs. With its changing mission, the F-22 was 
especially susceptible to changing requirements during the early stages of Spiral 2. 
While any large DoD weapon system program will be challenged with shifting 


requirements (called “creep’’), the addition of the air-to-ground capabilities to the existing 


28 At the end of DO 0002, LM Aero was commended for aggressively implementing 
process improvements that enabled increased performance and criticized (in award fee 
and past performance documentation) for excessive cost estimating. One of the 
responses LM Aero provided to this critique was that the DO 0002 excessive underrun 
was more due to changing requirements than poor budgeting. The following section 
looks at Spiral 2 requirements stability. 
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air-to-air mission of the F-22 presented especially challenging pressures on the 
modernization program to incorporate capabilities that had not been planned for the 
fighter. In addition to the changing mission of the program, the senior managers of F-22 
were fighting to retain the program funds as DoD sought dollars to pay for Global War on 
Terror operations. This resulted in capabilities being promised to senior DoD leadership 
and congress prior to any evaluations of the impact of incorporation into the subject 
Spiral. Although the Spiral development model was established to maximize the 
program’s ability to adjust requirements as required, it was never intended to facilitate the 


frequent changes in requirements that Spiral 2 experienced. 


As requirements analysis neared completion in January 2004, the requirements 
baseline was established for detailed design. This set of requirements was the basis for 
the DO 0002 UCA. In March 2004, shortly after DO 0002 was authorized, the first letter 
was written to LM Aero changing the major capabilities being developed for Spiral 2. In 
the summer of 2004, funding constraints and external political pressures forced the 
program to formally direct requirement changes to LM Aero four times between July and 
September. This instability in core Spiral 2 requirements, made the use of a PMB for DO 
0002 almost impossible. Every iteration of requirements changes led to a major 
rebaseline of the program plan. This, coupled with the cultural effect of knowing 
requirement changes were always being considered, led to lack of confidence in the PMB 
and, therefore, the EVM data on which it was based. Operating in this environment of 
constantly changing baselines yielded what was commonly referred to as “rolling 
baselines.”29 Rolling baselines were key indicators of the lack of program stability. 
Whether the instability was internal or external, the impact was the same: loss of 
confidence in EVM. Although the EVM process is designed with the intent of absorbing 
changes to the program baseline, when a program makes changes to its baseline as 
frequently as the F-22 Spiral 2 program did, the value provided by EVM begins to 


deteriorate as confidence is lost in the existing baseline. 


29 Similar to (and often a result of) requirements creep, this nomenclature describes a 
flexible baseline that lacks the stability essential for benchmarking performance 
measurements. 
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An additional problem with frequent rebaselines was the increased difficulty in 
identifying the history of the PMB updates. While the last five of the 32 ANSI/EIA 
EVMS criteria specifically address the need to control changes in PMB and other EVM 
baselines, the more frequently changes occurred, the more difficult it became to identify 
the relationships between current and former baselines. As changes continued to be made 
to the program baseline, whether in scope, resources, or schedule, it became more 
difficult to trace the relationship between the original baseline and its EAC and the 
current baseline and its EAC. This was critical for a program constantly being pressured 


to justify funding and provide measures of remaining effort. 


3: Timeliness of EVM Reports 


The EVM reporting cycle required in all Spiral 2 contracts was monthly. While 
Criteria #22 and #23 identify a minimum EVM reporting cycle of one month, most of the 
F-22 government users identified this reporting cycle as being insufficient to enable use 
of EVM as a management tool. LM Aero management and IPT leads had access to 
preliminary EVM data on at least a weekly basis; however, unless government managers 
established a trusting relationship with their LM Aero counterparts, they did not see any 
of this data until the formal EVM reports were delivered four to six weeks after the work 


was completed. 


The primary driver for this delay is the necessity of LM Aero to complete their 
monthly cost accounting before developing and delivering reports. If formal reports were 
provided to the government prior to the end of the accounting period changes may have 
been required at the end of the period. While many in the government would likely 
understand this situation and trade it for the ability to see the data earlier, it does expose 
LM Aero and the program leadership to potential problems if they make decisions based 
on preliminary data. Additionally, as changes occurred at the end of the accounting 
periods, there might be many who would lose confidence in the accuracy of the 


preliminary data. 
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Based on discussions with F-22 government managers, it was determined the 
desired approach was to establish a trusting relationship with the contractor manager to 
gain as much access as possible to preliminary EVM data. This enabled access to timely, 
if not completely accurate, data that supported management making decisions. Still this 
was an informal agreement and was completely based on the “good will” of the LM Aero 
manager. It also limited government managers in their ability to report emerging issues 
up their management chain. The first reason for this was because it was based on 
preliminary data that could change. The second concern was that it would “spoil” the 
relationship with the contractor manager who would be less likely to provide access to 


preliminary EVM data. 


C. SPIRAL 2 EVMS DATA ANALYSIS 


This section primarily addressed the second question posed within Methodology, 
Chapter I: “To what degree did the F-22 Spiral 2 implementation of EVM fulfill its role 
as a management control system for avionics software development?” To help answer the 
question, Cost Performance Reports (CPRs) directly related to Spiral 2 efforts were 
distilled into raw cost and schedule measurements. Then a Spiral 2 DO 0002 lifecycle 
EVM analysis was accomplished and compared to outputs from LM Aero reports. The 
comparison, in addition to the EVM data elements’ results and pertinent testimony from 


individuals involved, answered the question. 


In order to collect the data necessary, CPRs from the REDI contract modification 
known as Spiral 2 DO 0002 were gathered, the complete set dating from April 2004 
through January 2005. Using the raw numbers from the Format 2 reports, an Excel 
spreadsheet was created that took the lowest level IPT EVM data elements, summed them 
up through the IPT levels (as opposed to inputting data from the CPRs’ higher level 
IPTs), and generated trend lines for EVM data element analysis (e.g., BCWS, BCWP, 
ACWP, etc.). Please refer to Appendix I for the complete set of monthly reports. 
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1. CPR and Independent Analysis Comparison 


Beginning with the first CPR (April 2005), an interesting disconnect appeared 
between SV and CV. SV results put the program behind schedule by $31,000, while CV 
exhibited a cost underrun of $660,000—45 percent of the BCWP for that period. 
Conventional wisdom normally leads to the conclusion that behind schedule equates to a 
cost underrun, but usually at a somewhat similar measurement (e.g., SV = 10, CV = 12). 
A twenty-one-fold increase from SV to CV indicated either one of two conclusions, or 
some combination of both: efficiencies were occurring within one or more IPTs, or the 
original budget assumptions were grossly incorrect. Regardless, the EVM data was 


highlighting an area of concern for program management. 


Subsequent reports only heightened that concern. By midway through the CPR 
reporting cycle (August 2004), the CV indicated a $4.9M underrun—S6 percent of the 
ACWP. cumulative-to-date—while SV indicated a less unwieldy $63,000 behind 
schedule. That said, LM Aero’s Format 5 (Variance Analysis) for that period stated, 
“There [were] no current period, cumulative-to-date or at-completion variances which 
exceed[ed] thresholds in [August 2005]’s report.” Thus, even though the EVM data was 
indicating an area of concern regarding Spiral 2 DO 0002 performance, LM Aero’s 
analysis reports to the F-22 SPO indicated all EVM data elements were within tolerances. 
Not until the October 2004 report did LM Aero’s formal Variance Analysis begin to offer 


a reason behind the recurring, outsized CVs. 


In a similar case, this study’s independent EVM analysis uncovered an area of 
concern regarding the CPR reports’ data summarization. In the June 2004 report, the 
independent analysis revealed a relatively minor anomaly in reporting that continued into 
the final report (January 2005). Whereas the CPR in question reported BCWS 
cumulative-to-date as $6.775M for the entire program, the independent analysis reported 
the same data element as $6.863M, resulting in a delta of $88,000. Though small in 
relative size, this anomalous delta effected an investigation into the cause. The 
investigation found that when the May 2004 CPR’s BCWS cumulative-to-date 
($4.437M) was added to the June 2004 CPR’s BCWS current period ($2.428M), the 


result netted a $6.865M amount that should have corresponded to June 2004 BCWS 
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cumulative-to-date. Note this figure results in a minor $2,000 delta between the 
independent analysis’ same figures (easily explained by rounding error). As already 
disclosed, this error continued through subsequent CPRs, ranging in value from $82,000 


to $134,000. 


Successive study netted a similar anomaly. Beginning with the June 2004 CPR 
report, the independent analysis reported the BCWP cumulative-to-date data element as 
$6.694M; when compared to the actual CPR’s corresponding value of $6.562M, this 
resulted in a delta of $132,000. Again, the investigation found when the May 2004 
CPR’s BCWP cumulative-to-date ($4.570M) was added to the June 2004 CPR’s BCWS 
current period ($2.127M), the result netted a $6.697M amount that should have 
corresponded to June 2004 BCWS cumulative-to-date. Note that this figure results in a 
minor $3,000 delta between the independent analysis’ same figures (easily explained by 
rounding error). Like before, this phenomenon continued throughout the remaining 


CPRs, ranging in value from $114,000 to $166,000. 


In both BCWP and BCWS cases, the Mission Sys & SW (Avionics Systems 
Engineering Integration Team and Mission Avionics Software) and Modernization IPTs 
were the cost drivers for all affected CPRs. Investigation revealed the “What?” portion 


3? 


of the cause, but failed to uncover the “Why.” These inexplicable errors raise concerns 


regarding the validity and veracity of the reports given to the government. 


2. EVM Trend Analysis 


This study’s trend analysis benefited from hindsight. Given the facts known after 
the program concluded, the Spiral 2 DO 0002 program was clearly overestimated, a fact 
unknown to management until the effort was completed in early 2005. In theory, the 
EVM data elements should have reflected that fact. The following results—acquired 
from the independent EVM analysis—exhibited trends that supported, at least in part, the 
conclusion that the data elements did forewarn of problems related to program budget 


assumptions, which in turn translated to a probable exaggeration of BCWS and BCWP. 
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A top line analysis of current-period EVM data elements revealed that whatever 
the budget assumptions were that comprised the Spiral 2 DO 0002 undefinitized contract 
baseline, the work continued closely to plan throughout much of the program. The 
BCWS, BCWP, and SV lines in Figure 9 clearly denoted this. The figure also clearly 
indicated real costs lagged budgeted for all but one period of the program via the ACWP 
line (below both BCWS and BCWP) and resultant CV line (positive for all but one 
reporting period). These results lead to the conclusion that budgeted costs (represented 


by BCWS and BCWP) were overestimated relative to actual costs (i.e., ACWP). 
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Figure 9. F-22 Modernization REDI DO0002 Data Elements (Current) 


A top line analysis of cumulative EVM data elements supported the preceding 
conclusion (see Figure 10). Throughout the lifecycle of the program, BCWS and BCWP 
remained similar while ACWP lagged both. As a result, SV remained relatively flat and 
CV grew. These results lead to the conclusion that budgeted costs (represented by 
BCWS and BCWP) were overestimated relative to actual costs (i.e., ACWP). 
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Figure 10. F-22 Modernization REDI DO0002 Data Elements (Cumulative) 


The general question of whether EVM identified a disconnect between program 
budgeted costs and actual costs answered, did EVM identify any other leading indicators 
warranting management’s attention? To answer that question, the independent analysis 
looked at each EVM data element in terms of IPT totals (see Figures 11 and 12). As 
expected, the totals supported the suspicion of incorrect budgetary estimates. They also 
lead to the conclusion that much of the blame for the high CV lay with Mission Sys & 
SW. This particular IPT contributed the majority of cost to every data element analyzed. 
Knowing that, program management, theoretically, could have focused management 
control efforts on work occurring within that IPT. More to the point, the data provided 
by EVM indicated program instabilities in addition to identifying areas of the program 


which warranted closer attention. 
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Figure 11. 


Period) 


90 


F-22 Modernization Spiral 2 DO 0002 Data Elements Summed by IPT (Current 
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Figure 12. 


F-22 Modernization Spiral 2 DO 0002 Data Elements Summed by IPT 
(Cumulative-to-Date) 
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D. QUESTIONNAIRE ANALYSIS 


This section addresses the final question posed in this paper, “To what extent did 
the F-22 program management (Government and Contractor) use EVM products to 
manage avionics development efforts?” The method chosen to address this question was 
the use of a questionnaire. Following is an overview of the questionnaire and an analysis 


of the responses. 


1. Questionnaire Overview 


The questionnaire (see Appendix II) aimed to assess the perceived usefulness of 
EVM within a software development context. The questionnaire consisted of nine 
closed-ended questions falling into three distinct areas. The first area (Questions #1 and 
#2) focused on the respondent’s background by asking them to categorize themselves by 
government or contractor function(s) and area(s) of focus within F-22 development. The 
second area (Questions #3 and #4) of the questionnaire sought to establish the 
respondent’s level of interaction with EVM by looking for their frequency and method of 
use. The final area (Questions #5 through #9) built on the respondent’s interaction with 
EVM and sought their assessment of EVM. Questions in this area focused on the 
respondent’s perceived value and usefulness of F-22 EVMS and its data as well as a self- 
assessment of their knowledge of EVM and the 32 ANSI/EIA criteria. Those identifying 
themselves as being aware of the 32 ANSI/EIA criteria were asked to assess the F-22 


implementation of EVM with respect to these 32 criteria. 


The questionnaire was provided to a mix of 26 government and contractor F-22 
personnel. They included program managers, engineers, contracting officers, and 
financial managers who worked on the F-22 program. All government and contractor 
personnel directly managing Spiral 2 development were provided the questionnaire and 
responded. Additionally, other F-22 government managers not directly supporting Spiral 
2 or modernization were asked to respond to the questionnaire. All government 
personnel who were provided the questionnaire were assigned to the F-22 System 


Program Office (SPO). 
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2; Summary of Responses 


There were 25 responses to the questionnaire from both government and 
contractor F-22 personnel.30 Seventeen (68 percent) of the respondents were government 
SPO personnel. Although these 17 respondents only represent approximately 9 percent 
of the personnel assigned to the F-22 program office at the time of the study3!, they 
accounted for 100 percent of government personnel assigned to the Spiral 2 program. 
Additionally, all eight contractor respondents were Spiral 2 personnel. While the sample 
for the questionnaire was well-below a meaningful representation for the entire F-22 
program, it was a very good representation for the Spiral 2 program. Additionally, when 
looking at the area of the program supported by the respondents, 22 of the 25 responses 
(88 percent) were from personnel who support the modernization program. Because of 
the functional matrix organizational approach used by the F-22 program, it was difficult 
to determine the total number of personnel working on the modernization program; 
however, it was strongly believed by the authors that these 22 individuals represented a 


statistically significant number of modernization personnel. 


Figure 13 shows how the respondents’ support of program areas and elements was 
distributed for government and contractor personnel. When categorizing the program 
element(s) on which the respondents worked there was a majority of respondents who 
supported avionics. Because personnel support multiple areas and elements of the 
program, respondents were permitted to select more than one functional area and 


program element. 





30 One response was deemed invalid due to incomplete answers. 


31 Based on F-22 System Program Office headcount of 194 active duty military and 
DoD civilians provided in telephone interview with front office personnel on 13 
November 2006. 
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Figure 13. Distribution of Respondents 


When looking at some of the specific responses from the questionnaire, there 
were definite trends in some areas. Some of the most notable of these were with respect 
to the 32 ANSI/EIA criteria. Nineteen (76 percent) of the respondents stated they were 
not aware of the 32 criteria. Considering the importance of these criteria in determining 
whether an EVMS has been correctly implemented, it was surprising to see how many 
users were unaware of the criteria. Perhaps this was why EVM could have been 
implemented on Spiral 2 and had as many criteria be judged as marginal in their 
implementation. Another interesting trend with respect to the 32 criteria was, of the six 
respondents who said they were aware of the 32 criteria, all stated they believed EVM 
was being implemented on F-22 in compliance with these criteria. The binary (Yes/No) 
response to the question “Do you think your program implements its EVMS according to 
the 32 ANSI/EIA-748 Earned Value Management System criteria?” may have forced 
some respondents to go with the “predominate” response—after all, the question didn’t 
ask if their program EVMS meets all of the 32 criteria. It was still surprising for the 
authors to see 100 percent of these respondents respond in the affirmative to this question 


considering the authors’ findings on the criteria were not unanimous. 
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Additional trends were observed with respect to contractor versus government 
responses. The most notable was the increased frequency of EVM use on the part of 
contractor respondents versus government respondents (see Figure 14). Of the 17 
government personnel who stated some use of EVM, 15 (88 percent) of the respondents 
stated they used EVM less than once per week, while two stated they used it once per 
week. Contrasting this to the seven (87.5 percent) of the eight contractor respondents 
who all stated they used EVM at least once per week. Of these seven, two (25 percent) of 
the contractors stated they used EVM more than once per week. This disparity in 
frequency of use was not a surprise to the authors based on the concern raised by 
government personnel regarding the timeliness of EVM data. As discussed above, while 
contractor managers have access to preliminary EVM data less than one week old, the 
government only receives EVM data once it is provided in formal reports that are weeks 
later than the work being reported. This disparity in timeliness is the most likely driver 


for the disparity in frequency of use between the two organizations. 
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Figure 14. Frequency of EVM Use by Government and Contractor Respondents 
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3. Propositions 


The process of evaluating the responses to the questionnaires resulted in several 
propositions, each of which enabled further analysis of potential relationships and trends 
regarding EVMS usage and perceived value. Each of these propositions was evaluated 
individually using questionnaire responses to determine their validity. Although the 
results of the analysis of propositions are presented below, more detailed supporting data 


and values for the questionnaire responses and analysis is provided in Appendix II. 


a. Higher EVM Value Results in More EVM Usage 


The first proposition was that an individual who places higher value on 
EVMS as a management tool will use EVMS more frequently. Of the 25 respondents, 17 
(68 percent) stated they viewed EVMS as having moderate or lower value (three or less 
on a scale of one to five). Meanwhile, there were 16 (64 percent) respondents who stated 
they used EVMS more than once per month. When testing the relationship between these 
two variables, there was not enough statistical evidence to accept this proposition with 95 
percent certainty. However, there was enough of a relationship between these variables 
to state that, having 90 percent certainty, the more value an individual placed on EVMS, 


the more they used the tool. 


This proposition, supported by the questionnaire results32, provided some 
measure of confidence in the validity of the questionnaire. It would have been counter- 
intuitive for there to not have been a relationship between perceived value and frequency 


of use. Any tool that is truly believed to be valuable should be used more often. 


b. Higher EVM Usefulness Results in More Usage of EVM in 
Managing 


The second proposition was that individuals who had an opinion of higher 
usefulness for EVMS would use it more as a management tool versus just a reporting 
32 Per statistical analysis, propositions are either “rejected” or “not rejected.” Failure 


to reject a proposition does not necessarily make it “true.” For simplicity, the authors 
refer to propositions that were “not rejected” as being true. 
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tool. Although EVMS is intended to be used as a management tool, 11 (44 percent) of 
the F-22 respondents either solely or predominately used EVMS to receive and report 
cost and schedule status. Twelve (48 percent) of the respondents stated they used EVM 
data either equally as a management and reporting tool or predominately as a 
management tool.33_ As discussed above, many times the timeliness of the EVM reports 
limited the utility of EVM as a management tool. This proposition sought to identify any 
linkage between respondents’ perceived usefulness and more meaningful usage of the 


data. This proposition was also supported by the responses with 90 percent certainty. 


The difficulty in interpreting this proposition was which variable was the 
driver. That is, did method of use drive perceived value (i.e., a management tool is likely 
to be deemed more “useful” than a reporting tool) or did perceived usefulness drive 
method of use (i.e., a meaningful source of data will be used for managing instead of 
merely reporting)? This question could not be answered by the data collected in the 
questionnaire; however, it was important enough merely to confirm there was a linkage 
between these two responses. Regardless of the driver, the relationship existed in the 
questionnaire responses and pointed to the important—perhaps essential!—linkage 


between the two variables. 


Cc. Higher EVM Knowledge Results in Higher EVM Value 


The final proposition was that individuals who identified themselves as 
having more knowledge regarding EVM would place a higher value on EVM. The 
rationale behind this proposition was that individuals who knew more about the data and 
application of EVM would place more value on its ability to be used as a management 
tool. Based on the results of the questionnaire, however, this was not the case. 
Regardless of the level of certainty, there was nothing in the results to support this 
proposition. This indicated respondents determine their value in EVMS via other means. 
One likely source would be their own “return” on using the EVM products. This is just 


one potential driver and the question of where the value was derived was not specifically 


33 Three respondents did not identify any method of use in their questionnaire 
responses. 
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addressed in the questionnaire. If this questionnaire is representative, it does indicate that 
merely teaching and training about EVM is not sufficient to convince managers to value 


the tool. 


While the questionnaire results could be broken down and analyzed in any 
number of additional ways, the authors believe these propositions are the most relevant to 
answer the question of whether EVM is used by F-22 personnel to manage the program. 
The following final chapter will address the authors’ findings regarding this and other 


questions posed in this paper. 


E. SUMMARY 


This concludes Chapter IV, F-22 EVMS Environment. The next chapter, 
Conclusion, presents a condensed synopsis of this research project’s outcome, includes a 
brief discussion on limitations with respect to the research project, and makes final 


recommendations to the EVMS where necessary. 
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Vv. CONCLUSION 


A. SUMMARY OF FINDINGS 


The intent of this study was to academically appraise the F-22 program’s use of 
EVM in managing avionics software development within the Spiral 2 REDI contract 
effort, Delivery Orders 0002 and 0019. Achieving this goal involved performing and 
reporting the results of 1.) a detailed, data-supported evaluation of how the program 
meets each of the 32 ANSI/EIA EVMS criteria; 2.) interviews with subject-matter 
experts; 3.) a statistical questionnaire conducted with government and contractor 
personnel involved in F-22 software development. In order to facilitate the assessment, 
there were three questions asked by the authors. The following are the findings and 


recommendations associated with each question: 


1. How Closely Did Implementation of EVM Follow ANSI/EIA Criteria? 

a. Findings 
Answering “How closely did the F-22 Spiral 2 implementation of EVM 
follow the criteria outlined in ANSI/EIA-748-A-1998 Earned Value Management 
System?” relied mainly on a careful assessment of how EVM implementation for Spiral 2 
supported the 32 ANSI criteria. Based on the review of the objective of each criterion, 
the F-22 self-described implementation of the criteria, and interviews, data, and direct 
observation an assessment was made for each of the 32 criterion. These assessments 
were solely those of the authors and do not represent an official government position on 
the F-22 implementation of their EVMS. A summary of the 32 criteria assessments are 
as follows: seven excellent criteria, 15 satisfactory criteria, five marginal criteria, and one 
insufficient criterion, and four inconclusive criteria. With 22 (69 percent) of the 32 
criteria assessed as satisfactory of better, the F-22 implementation of EVM was fairly 
strong; however, improvement of the F-22—and follow-on programs’—use of EVM 
during software development strongly depends on a discussion of the marginal and 


insufficient criteria. 
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Although there were five criteria deemed marginal, three of the five 
criteria were in one of the five criteria groups. Three of the ten Planning, Scheduling and 
Budgeting criteria were marginal. Another criterion could not be assessed sufficiently for 
a rating due to lack of supporting data. This trend was consistent with discussions and 
direct observations regarding the environment of F-22 development. As discussed in 
Chapter IV, F-22 EVMS Environment, the use of undefinitized contracts and 
requirements instability made it extremely difficult to adequately plan, schedule and 


budget for Spiral 2 efforts. 


Criterion #26, Implement managerial actions, was the only criterion to be 
assessed as insufficient. This was largely based on the apparent absence of managerial 
actions taken on DO 0002 even as it had excessive variances and progressed toward a 
significant cost underrun. While it seems counter-intuitive to criticize a cost underrun, if 
corrective measures had been taken earlier, the program may have been able to capitalize 
on these additional funds before DO 0002 work was completed. The potential reasons 
for the failure of the program to take managerial actions are discussed in the following 
section. Suffice it to say, it does little good to collect EVM data if the program chooses 


not to use it. 


b. Recommendations 


If EVM is to be successfully used by F-22 personnel to manage avionics 
development, research data suggests that program managers dedicate themselves towards 
moving away from the use of undefinitized contracts. While this is easier stated than it is 
accomplished, based on the goals and intent of many of the ANSI/EIA EVMS Criteria, 
EVMS will not be fully useful as a management tool to F-22 as long as it is used in 


concert with undefinitized contracts. 


A process for ensuring more disciplined control of requirements should be 
sought after. Research showed that stable requirements are critical for reliable, maturing 


EVM products. Group 2 of the ANSI/EIA EVMS Criteria focus on Planning, Scheduling 
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and Budgeting. It is difficult if not impossible to meet the intent of the criteria outlined 


under this area if requirements are not well-defined. 


Zz. To What Degree Did Implementation of EVM Fulfill Its Management 
Control Role? 


a. Findings 

In Chapter II, Literature Review, there were historical indicators that the 
F-22 program either chose to ignore or did not have confidence in the message that EVM 
was providing during EMD. After collecting and analyzing Spiral 2 CPR data along with 
assessing responses to the EVM Questionnaire, the authors’ conclusion was there are still 
problems within the F-22 program in this area. One driver for the lack of use of EVM as 
a management tool is lack of confidence in EVM. It could not be determined whether 
this was due to unique F-22 problems (e.g., rolling and/or immature PMB) or to a more 
general lack of perceived EVM value. Both questionnaire results and interviews did 
indicate less use of EVM on the part of the government compared to the contractor. This 
could have been due to the significant time delay between when contractor managers first 


see EVM data and when government managers finally see it. 


When considering managerial tools for use, the timeliness of the data is 
critical. However, the bottleneck in this instance appeared to be LM Aero business 
practices, not the EVMS. Streamlining these business practices should be considered 
paramount when attempting to shorten the timeline between EVM data collection and 


reporting. 


The analysis of Spiral 2 EVM reports did indicate some errors in 
accounting/reporting; however, none of these attained a magnitude sufficient enough to 
bring into question the usefulness of the EVM reports. While there should be questions 
asked regarding these errors to ensure accurate, trustworthy reporting, the authors did not 
believe these errors were systemic in nature. The Spiral 2 data, like the EMD data, had a 
story to tell. As discussed above, there was a lack of action in response to this data. This 
did not appear to be a result of some malfunction of the F-22 EVMS, but rather inaction 


on the part of program managers. 
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b. Recommendations 


If EVM is to fulfill its role as a managerial tool in the F-22 program, 
research concludes that steps should be taken to ensure government access to EVM data 
in a timelier manner. Options should be explored and steps taken to leverage current 
customer-oriented industry practices that employ business procedures to shorten response 
timelines (including customer reports). Data suggests that investment into such process 


optimization will allow a more complete utilization of any EVMS. 


3. To What Extent Did F-22 Management Use EVM to Manage Avionics 
Development? 


a. Findings 

The EVM Questionnaire aimed to assess the perceived usefulness of EVM 
within a software development context and to assess F-22 managers’ perceived value of 
EVM with respect to their management duties. Interviews supplemented the data from 
the questionnaire, figuratively filling in the questionnaire’s information gaps uncovered 
throughout the course of the research project. Combining these assessment tools with 
those used for the other areas of research provided a full picture of how F-22 avionics 
managers use EVM. Questionnaire results indicated there is a somewhat low perceived 
value for EVM. Additionally, there were indications that EVM is used more as a tool for 
receiving and reporting cost and schedule status versus using it as a tool for proactively 
managing the development effort. Based on interviews conducted with several F-22 
avionics managers, this becomes truer as software development efforts progress from 
design towards coding and into integration and testing. The more defined the tasks, the 
more confidence existed in EVM reports. As programs such as Spiral 2 moved toward 
less defined tasks, the less value was placed on EVM and the less it was used as a 


management tool. 
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b. Recommendations 


Results of the data analysis and questionnaire analysis indicate a clear 
trend of F-22 managers either ignoring EVMS data or not using it enough for it to be 
considered an active management tool. The authors submit this is largely due to the 
perceived value of F-22 EVMS products. Regardless of whether it’s driven by the 
managers’ knowledge of F-22 EVMS flaws (as pointed out in this paper) or by a more 
generic lack of appreciation for EVMS as a management tool, if the F-22 software 
development programs are going to take advantage of the benefits of EVMS, the program 
must first establish confidence in it. Some of this confidence can be gained by 
implementing earlier recommendations that would make available more reliable, stable 
and timely data to government and contractor managers. Additional confidence could be 
gained by providing focused training for software development managers on the strengths 
and weaknesses of EVMS with respect to software development. This would enable 
managers to become more knowledgeable regarding how program decisions (such as 


proceeding with UCAs) affect the value and usefulness of EVM. 


EVMS is particularly useful in the early stages of software development; 
however, research indicates it becomes less reliable and useful as the software 
development program progress through integration and test. Although a suggestion of a 
another software management tool is beyond the scope of this research project, the F-22 
program should leverage off of knowledgeable resources such as the Carnegie Mellon 
Software Engineering Institute and the USAF Software Technology Support Center to 
explore the use of other management tools to either augment or replace EVM during the 


latter stages of software development. 


4. Summary 


The cumulative effect of assessing the criteria, evaluating the Spiral 2 EVM data 
and assessing questionnaire responses along with interview statements indicated that in 
the case of Spiral 2, there is a limited use of EVM by F-22 personnel in managing 


avionics development efforts. It is necessary for the program to evaluate the reasons 
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behind this conclusion, whether it was lack of perceived value, lack of understanding of 
its function, or lack of confidence in the data. Regardless of the reason, data suggests 
that the F-22 program management should either take steps to address better use of EVM 
or identify other management tools in its stead. Either of these actions will help assure 


the program avoids repeating performance challenges endured during EMD. 


B. LIMITATIONS 


Within the context of this project, every effort had been made towards a full 
assessment of the F-22 avionics program’s EVM implementation, replete with all 
essential information. Unfortunately, reality dictated something less than full collection 
of required data. Regardless of completeness, the very information collected during the 
course of this academic pursuit may represent an outcome approximating, rather than 
equal to, the reality of the situation. As such, several limiting factors with regard to this 


particular study must, in the name of full academic disclosure, come to light. 


To begin with, the nature of cutting-edge weapons systems like the F-22 dictated 
that certain information remained, in some cases, privileged. Thus, during the course of 
this study’s investigative phase some information collected was accompanied with 
instructions to disseminate in a general manner, as opposed to reporting details that may 
or may not have altered this project’s outcome or its readers’ opinions. This requirement 
also pertained to first-hand knowledge that the authors may or may not have added to this 


paper’s findings. 


Secondly, and in a much-related matter, the nature of government and defense 
contractor relations necessitated a certain amount of caution during the course of 
interactions between representatives of the two parties. Unfortunately, though both 
parties make sincere efforts to maintain a mutually beneficial environment of information 
sharing, in the past both parties have failed to ensure that the sharing of sensitive 
information (e.g., Government Budgets, “Insider” schedule and cost reports) does not 
harm the party sharing that potentially inflammatory data. As a result, this project’s 
investigative efforts fell short of acquiring some data initially requested from the 


contractor. 
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Finally, the analysis phase of this project has uncovered issues regarding the 
EVM Questionnaire. The first issue dealt with the small and specific population size that 
the questionnaire analysis depends upon. This introduced biases rooted in program 
culture and program-specific training, even within the context of the F-22 SPO / LM 
Aeronautics: F-22. Thus, should problem areas with respect to EVM implementation 
become visible, an analysis of questionnaire data may or may not uncover systemic 


causalities that “perfect” information might identify. 


The second questionnaire issue involved the discrete values assigned to the EVM 
Questionnaire queries. For example, Question #3’s possible answer began with “Never” 
and proceeds to “Once a Month”. Post-submission analysis indicated that a substantial 
number of personnel, especially those not identifying “Financial Management” as their 
“current role”, actually would answer somewhere in-between these given choices (i.e., 
more than “Never” but less than “Once a Month”). Thus, though careful thought went 
into the questionnaire design and implementation, the discrete choices given to 


respondents appeared to have affected the questionnaire analysis. 


Limitations aside, the information gathered and interpreted provided sound data 
collection, results, and recommendations given realistic constraints that would affect any 
similar undertaking. An honest appraisal of research limitations should not preclude the 
fact that perfect information almost never avails itself to those who seek it. That said, the 
remainder of this chapter reviews the highlights of this study and_ presents 


recommendations for consideration. 


C, AREAS FOR FUTURE STUDIES 


Although the authors sought to extensively assess the use of EVM in the F-22 
program, several areas for further study were identified on this topic. The first was in the 
area of applicability of EVM to the latter stages of software development. As software 
development progresses from well-defined tasks such as development of design 
documents, writing code and developing test plans to less defined tasks such as recode 
and retest, EVM becomes less useful as a management tool. Research in the area of 


potential approaches for measuring value in this less-defined regime would be valuable. 
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Spiral 2 has not yet completed its DO 0019 effort which is largely comprised of these 
“less-defined” tasks. A study of how EVM implementation was employed at the 
conclusion of this delivery order would be useful in identifying potential alternatives or 


variations to current EVM implementation. 


F-22 is plagued with reliance on undefinitized contractual actions. Considering 
many of the challenges facing F-22 are not unique, it is reasonable to assume excessive 
use of UCAs is also not unique. Research into the trends and impacts of this type of 
contractual approach might provide further information to managers and contract 
officers. UCAs are not wrong; however, their use does come with consequences. 


Research that quantified these consequences would be useful. 


Another challenge identified for the F-22 program was the timeliness of EVM 
reports to government managers. This is also not an issue unique to the F-22 program. 
As the authors discussed, there are valid reasons behind the delay in providing trusted 
EVM data to the government; however, there are potential solutions to these 
challenges.34 Research into the major drivers for this reporting delay, potential solutions, 
and costs associated with these solutions could provide DoD some viable options to solve 


this issue and make EVM as valuable to the government as it is to the contractor. 


34 See Recommendations for Research Question #2. 
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APPENDIX A — SPIRAL 2 DO 0002 COST AND EVM DATA 
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APPENDIX B-—- EVM QUESTIONNAIRE 


Please answer the following questions by circling your selection. 


QUESTION #1 Using the options provided below, circle the function that best 
describes your current role within the F-22 program: 


Government Employee 








Contract Engineer Financial Program Other 

Manager Manager Manager 
Contractor Employee 

Contract Financial Program 

Manager Engineer Manager Manager a 


Other: 





QUESTION #2 Using the options provided below, identify the areas within the F-22 
program you have working knowledge of (circle all that apply): 





Airframe Avionics Support Propulsion Other 
Systems 
Other: 
EMD Modernization Production Sustainment Other 
Other: 





QUESTION #3 Using the scale provided below, identify how often you use or are 
exposed to some form of EVMS: 


Onsa nar More Than Ones per More Than 
Never Sain Once per Weck Once per 
Month Week 
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QUESTION #4 Using the options provided below, identify which best describes how 
you utilize your program’s EVM information: 


Reporting Tool Equal Parts Management 
Reporting with (some) Reporting and Tool with Management 
Tool Only Management Management (some) Tool Only 
Uses Tool Reporting Uses 


QUESTION #5 Using the options provided below, please assess the value of your 
program’s EVM information: 


1 2 3 4 5 
i 
Very Little Value Very High Value 


QUESTION #6 Using the scale provided below, please self-assess your knowledge of 


EVM: 
1 2 3 4 5 
i 
What’s EVM, again? Iam an EVM master! 


QUESTION #7 Are you aware of the 32 criteria set forth by ANSI/EIA-748 for 
assessing Earned Value Management Systems? 


YES NO 


If you answered “YES”, please proceed to question #8. 
If you answered “NO”, please proceed to question #9. 


QUESTION #8 Do you think your program implements its EVMS according to the 32 
ANSIJ/EIA-748 Earned Value Management System criteria? 
YES NO 


QUESTION #9 Using the options provided below, please describe the usefulness of 
your program’s EVMS: 


1 2 3 4 5 
—_ 
Not at All Useful Very Useful 
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APPENDIX C —- EVM QUESTIONNAIRE RESULTS 


A. RAW QUESTIONNAIRE RESPONSE DATA 





Q Focus Potential Responses Actual Responses Totals 
14| 7| 8/15/11] 9/12] 1| 4 3} 6) 17) 21) 22| 23) 24) 25] 26| 27| 28) 29] 30) 31| 32| 33 


1 Govt K Mngr 
Govt Eng 
Govt FM 
Govt PM 

Govt Other 

Ktr K Mngr 
Ktr Eng 

Ktr FM 
Ktr PM 
Ktr Other 

2 Airframe 
Avionics 
Support 

Propulsion 

Other 
EMD 
Mod 
Prod 
Sust 
Other 
3 Never 
Once per Mo 
Frequency More than 1/Mo 
Once per Wk 
More than 14k 
4 Report Only 
Report & Same Manage 
Method of Use Report/Manage 
Manage & Some Report 
Manage Onl 
5 1 (Very Little) 


Function 





NwotNoalororol-aw~sam = 






Area 





=n = 
on 


9 


Value 3 





5 (Very High 
6 1 (??) 
2 


Knowledge 3 
4 

5 (Master 
a Yes 
32 Criteria No 
8 Criteria Yes 
Implementation No 

9 1 (Not) 
2 





Usefulness 3 1 Aye 1 1] 1 1 1 1 
4 1] 1 1] 1 1 1 1 


5 (Very) 1 
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B 

















QUESTIONNAIRE STATISTICAL ANALYSIS RESULTS 








Contract Manager 





44 





20 0.00) 


20 0.00) 20) 0.00) 


#0IVi0! 


0.00) 


Criteria 
Frequenc Method of Use Yalue Knowledge. 32 Criteria ___implementation _Usefulness 

‘Sample Mean] 34 26 239 33 18 10 3.0 

Equivalent Recponce | Morethan tfMo | ReportiManage 30 30 No Yee 30 
Sample S.D] 1.00 0.83 413 0.84 O44 0.00 0.89 

By Function, All 
not Frequency Method of Use Value Knowledge. 32 Criteria Criteria Implementation Usefulness. 
Number Respondents SD. SD. SD. SD. SD. SD. SD. 











0.00) 











Program Manager 7 28% 38 13 33 08 23 i 33 i 16 05 10 20 30 08] 
Other i 4% 20 0.09] 40 0.09] 20 0.09] 40 0.09] 20 o.00[__#DIv/0! 0.09] 20 0.09] 
By Function, Government 
Kot Frequency Method of Use Value Knowledge 32Criteria Criteria Implementation Usefulness 
Number__| Respondents p S30. p SD. y SD. y SD. L SD. v SD. L SD. 
Contract Manager i 2.0) 0.00] 20) 0.00] 10) 0.00] 20) 0.00] 20) 0.00] #DIViO! 0.00 20) 0.00] 
Engineer a 7 YT | 22) 04) 18} 08] 22] 08] 3.0) 07] 2.0) 0.0] #DIvior #DIVO! 26) 05) 
Financial Manager 7 z 34 a7 25 05) 33 a7 33 i 16 05 10 20 37, 10 
Program. Manager 3 x 23) 06) 3.3) 12] 23) 15 32) 06) 17) O86) 10) #DIViO! 27 12) 
Other 1 20 0.00] 40 0.09] 20 0.09] 40 0.09] 20 Coo] #oIv/O! 0.09] 20) 0.00) 
By Function, Contractor 
not Frequency Method of Use Value Knowledge 32 Criteria Criteria Implementation Usefulness: 











Number Respondents SD. SD. sD. SD. sD. sD. SD. 
Contract Manager er eS 7 ee eT 























=] | 
Program Manager 4 45) 06) 33) 05) 33) 05) 4.0) 0.0) 15) 08) 1.0) 0.0) 33) 05) 
Other 0 y 
By Area, All 
not Frequency Method of Use Value Knowledge 32 Criteria Criteria Implementation Usefulness: 
Number Respondents p SD. p SD. m SD. m SD. m SD. m SD. m SD. 
Airframe 6) 24% 238) 0.98] 238) 0.98] 27 121] 35) 0.84) 18) 0.41) 10) #DIviot 3.0) 1.10] 
Avionics 21) 84y 31 A} 27) 0.3) 23) A 33) 07) 13] OF 1.0) 0.0) 3] 0.3) 
Support ai 44%) 35) 0.3) 28) 07) 3.0) A 35) 07) 18 05) 1.0) 0.0) 3.2) 1.0) 
Propulsion. 3 12% 2.0) 0.0) 2.0) 0.0) 2.3] Os] 3.3] 12) 20) 0.0) #DIVIO! #DIVIO! 20) 0.0 
Other 2 iY 3.) 0.00) 25) 0.71} 35) 0.71} 45) 0.71} 1.0) 0.00) 1.0) 0.00) 35) 0.71} 
By Area, All 
not Frequency Method of Use Value Knowledge 32Criteria Criteria Implementation Usefulness: 











[Less than One # wk 


[More than Once ? wk 


By Method of Use 


Number 











|Beport On 

Report & Some Manage 
ReportiManage 
Manage & Some Report 
[Manage Oni 


By Value 


Jo] Joo}co fro 


30. 





30. 30. 


3.0. 











‘Average or below 
[Above Average 


By Knowledge 







Kot Frequency Value Knowledge 32Criteria Criteria Implementation Usefulness 
Respondents S.D. S.D. $.D. sD. SD. SD. 
8% 20 0.00) 15) on 3.0) 0.00) 20) 0.00] #DI¥i0L #DI¥iO! 25) 0.71) 
23) 07] 25) ES) ec) 5) ET) 7 23) a8] 
33 1.0) 38 07 35) 08 16) 05) 10) 0.0) 38) 07) 
28] 10 25) ES) Te |) a 1 | 10 0.0) 30) 12] 
00 00 00 00 00 0.0) 0.0) 0.0) 0.0) 0.0) 0.0) 0.0) 
not Frequency Method of Use Knowledge 32 Criteria Criteria Implementation Usefulness 





687 





‘Average or below 
Above Average 


By 32 Criteria 







Kot 


$0. 
29 19) 


Frequency 


25) 10] 


Method of Use 












$0. 


riteria 


Criteria Imp 










Usefulness 





Respondents 








By Usefulness 


not 





3.0. 


Frequency 










Criteria Implementation 





30. 


Usefulness 





Respondents 








[Average or below 











3.0. 


Frequency 





SD. 


#DIVIO! ‘#DIVIO! 


32.Criteria 


8.0. 


Criteria Implementation 

















Proposition Tests G@re.siced Two-sample F-test 


Pz: Higher knowledge results in higher value 


131346024 
1152783683, 
osiz84941 





$0. 














Pz: Higher usefulness results in more management use. 


observed difference = 
a: 

df= 

tdist = 

2 

Rejection Region 
Conolusion = 





0s 


23 
131946024 
0.866204687 
0.490024041 
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$0. 


















Pz Higher value results in higher frequency 


observed difference = 08 
ae 

df= 23 

tdist = 131946024 

5,2, 0.972775185 

Rejection Region = 0,550312454 





Conclusion = 


30. 























Mod 22| 32) 1.0) 27) 08) 3.0) 12 33) 03) 12 05) 1.0) 0.0) 32) 0.3) 
Prod 10) 23] 03) 26 0.3] 28) 0.3 34 1.0) 18) o4 10) 0.0) 23] 0.3] 
Sust 3] 33) 0.3) 25) 08 23) 0.3) 37) 07) 17] 05) 1.0) 0.0) 3) 08} 
Other 1] 3.0) 0.00) 2.0 0.00) 3.0] 0.00) 5.0] 0.00) 1.0) 0.00) 1.0) 0.00) 3.0] 0.00) 
By Frequency of Use 
not Method of Use Value Knowledge 32 Criteria Criteria Implementation Usefulness: 
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